single-view and multi-view planar models for dense ...csadc/lpm17/slides/lpm2017slides-civera… ·...
TRANSCRIPT
![Page 1: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/1.jpg)
Single-View and Multi-View Planar Models for Dense Monocular Mapping
Alejo Concha, José M. Fácil and Javier Civera
SLAMLab – Robotics, Perception and Real-Time Group Universidad de Zaragoza, Spain
International Workshop on Lines, Planes and Manhattan Models
for 3-D Mapping (LPM 2017) September 28, 2017, IROS 2017, Vancouver.
![Page 2: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/2.jpg)
Index
• Motivation • Background (direct mapping)
• Dense monocular mapping.
• Superpixels in monocular mapping • Superpixel triangulation. • Dense mapping using superpixels. • Superpixel fitting.
• Learning-based planar models in monocular mapping
• Data-driven primitives • Layout • Deep models
• Conclusions
![Page 3: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/3.jpg)
Motivation
• The scene model is limited in feature-based monocular SLAM.
• Our goal: Dense mapping from monocular (RGB) image sequences
![Page 4: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/4.jpg)
Background: Dense Monocular Mapping High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Sparse/Semi-d.
Dense
![Page 5: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/5.jpg)
Dense Monocular Mapping: Low Texture
High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Dense
![Page 6: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/6.jpg)
Superpixels (mid-level)
High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Sparse/Semi-d.
Dense
Superpixels
Dense + Sup.
• Image segmentation based on color and 2D distance.
• Decent features for textureless areas • We assume that homogeneous color
regions are almost planar.
![Page 7: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/7.jpg)
High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Dense
Dense Mapping: Low Texture
![Page 8: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/8.jpg)
Semi-dense Mapping: Low Texture
High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Sparse/Semi-d.
![Page 9: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/9.jpg)
2D Superpixels: Low Texture
High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Superpixels
![Page 10: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/10.jpg)
Superpixel Triangulation
H
Multiview model: Homography (ℎ = 𝐾(𝑅 + 𝑡𝑛/𝑑)𝐾−1)
Error: Contour reprojection error (ɛ)
Montecarlo Initialization: For every superpixel we create several reasonable {𝑛, 𝑑} hypothesis and rank them by their error.
![Page 11: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/11.jpg)
Superpixel Triangulation
Multiview model: Homography (ℎ = 𝐾(𝑅 + 𝑡𝑛/𝑑)𝐾−1)
Error: Contour reprojection error (ɛ)
Mapping: Minimize the reprojection error.
H
![Page 12: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/12.jpg)
Superpixels in low-textured areas
High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Superpixels
![Page 14: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/14.jpg)
Dense + Superpixels
![Page 15: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/15.jpg)
Dense + Superpixels
High Texture Low Texture
Accuracy Density Cost Accuracy Density Cost
Dense + Sup.
![Page 16: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/16.jpg)
PMVS (high-gradient pixels) Dense (TV-regularization)
Superpixels PMVS + Superpixels Dense + Superpixels
Video (input)
Dense + Superpixels (5 centimetres error!)
Alejo Concha and Javier Civera. Using Superpixels in Monocular SLAM. ICRA 2014
Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(8):13621376, 2010.
Based on Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. Dtam: Dense tracking and mapping in real-time. In Computer Vision (ICCV), 2011 IEEE International Conference on, pages 23202327. IEEE, 2011.
Alejo Concha, Wajahat Hussain, Luis Montano and Javier Civera, Manhattan and Piecewise-Planar Constraints for Dense Monocular Mapping, RSS 2014.
![Page 17: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/17.jpg)
Fitting 3D Superpixels to Semi-dense Maps
• TV-regularization is expensive, GPU might be needed for real-time. • Semidense mapping and superpixels is a reasonable option cheaper than
TV-regularization (CPU) and with a small loss on density. • Having a semidense map superpixels can be initialized via SVD more
accurately and at a lower cost. • LIMITATION: We need parallax!!
Code at https://github.com/alejocb/dpptam
![Page 18: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/18.jpg)
Data-driven primitives (mid-level)
.
Feature discovery on RGB-D training data.
Extracts patterns that are consistent in D and discriminative in RGB
At test time, from a single RGB view we can predict mid-level depth patterns.
![Page 19: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/19.jpg)
Multiview Layout (high-level) (a) Sparse/Semidense reconstruction. (b) Plane normals from 3D vanishing points (image VP, backprojection, 3D clustering). (c) Plane distances from a sparse/semidense multiview reconstruction. (d) Superpixel segmentation, geometric and photometric feature extraction. (e), (f) Classification (Adaboost)
![Page 21: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/21.jpg)
Superpixels, Data-Driven Primitives and Layout
• NYU dataset, high-parallax sequences
![Page 22: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/22.jpg)
Superpixels, Data-Driven Primitives and Layout
• NYU dataset, low-parallax sequences
![Page 23: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/23.jpg)
Single-View Depth Prediction • Several networks already
exist (Eigen14, Eigen15, Liu15, Liu15, Chakrabarti16, Cao16, Godard16, Ummenhofer 16…)
![Page 24: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/24.jpg)
Deep Learning Depth vs. Multiview Depth
Deep Learning Depth Multiview Depth
Fairly accurate in all pixels Very accurate in high-gradient pixels,
inaccurate in low-gradient ones
Fairly accurate for single view Very accurate for high-parallax motion,
inaccurate for low-parallax one
No model for the error Good model for the error
Approximate scale 3D reconstruction up to scale
Errors depend on the image content Errors depend on the geometry
![Page 25: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/25.jpg)
Fusing depth from deep learning and multiple views
• The fusion is not trivial. • No uncertainty for CNN
depth. • Errors come from different
sources.
• Our assumption is • In general, deep learning depth
is more accurate • Multiple view more accurate
for high texture - high parallax
![Page 26: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/26.jpg)
Results
• The error of deep learning depth is ~50% lower than multi-view one. • Our fusion reduces the error ~10% over the deep learning results. • The scale invariant metric shows that our fusion fixes the structure. • Deep depth generalizes well (Eigen15 was trained on NYU but is accurate
on TUM)
![Page 27: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/27.jpg)
Conclusions (no free lunch!)
Point-based features (low-level)
High accuracy iff ↑texture and ↑parallax.
Superpixels (mid-level)
High accuracy iff ↓texture and ↑parallax.
Data-driven primitives (mid-level)
Fair accuracy for → ↑ texture and ↓parallax.
Not fully dense.
Layout (high-level)
Fair accuracy even for ↓texture and ↓parallax.
Assumes a predetermined scene shape.
Deep learning (mid/high-level)
Fair accuracy even for ↓texture and ↓parallax.
Fully dense.
More general.
![Page 28: Single-View and Multi-View Planar Models for Dense ...csadc/LPM17/slides/LPM2017slides-civera… · Single-View and Multi-View Planar Models for Dense Monocular Mapping Alejo Concha,](https://reader033.vdocuments.us/reader033/viewer/2022042316/5f04e9fd7e708231d4105603/html5/thumbnails/28.jpg)