(crowdsourced) image-based 3d modeling - microsoft.com · 3d modeling pipeline 13 images scene...
TRANSCRIPT
![Page 1: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/1.jpg)
(Crowdsourced) Image-based 3D Modeling
Johannes Schönberger
Senior Scientist, Microsoft
MR & AI Zurich Launch, 2019
1
![Page 2: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/2.jpg)
Image-based 3D Modeling
Images
2
3D Model
![Page 3: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/3.jpg)
Applications
3
Archiving, Visualization Mapping: Localization & Navigation
Autonomous RobotsMixed Reality
![Page 4: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/4.jpg)
4
Cameras Everywhere
![Page 5: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/5.jpg)
• Statistics 2018
• Trillions of images taken worldwide
• Billions uploaded to online services
Crowdsourced Imagery
5
*InfoTrends
![Page 6: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/6.jpg)
Crowdsourced Imagery
6
• Diverse
• Unstructured
![Page 7: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/7.jpg)
Challenges
• Robustness
• Completeness
• Accuracy
• Efficiency
• Scalability
• Privacy
7
2000 2005 2010 2015 2020N
um
ber
of
imag
es
![Page 8: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/8.jpg)
Pioneering Work
8
Pollefeys et al., 1997, …
Visual Modeling from a Handheld Camera…
![Page 9: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/9.jpg)
Pioneering Work
9
N. Snavely, S. Seitz, and R. Szeliski, 2006
Photo Tourism: Exploring Photo Collections in 3D
F. Schaffalitzky and A. Zisserman, 2002
Multi-view Matching for Unordered Image Sets,or How Do I Organize my Holiday Snaps?
S. Agarwal, Y. Furukawa, N. Snavely,I. Simon, B. Curless, S. Seitz, and R. Szeliski, 2009
Building Rome in a Day
J.-M. Frahm, P. Fite-Georgel, D. Gallup, T. Johnson,R. Raguram, C. Wu, Y.- H. Jen, E. Dunn, et al., 2010
Building Rome on a Cloudless Day
![Page 10: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/10.jpg)
Industry Examples
10
3D Photo Tours
![Page 11: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/11.jpg)
Industry Examples
11
Microsoft ASA(Azure Spatial Anchors)
![Page 12: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/12.jpg)
3D Modeling Pipeline
12
Images Sparse Model Dense ModelScene Graph
![Page 13: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/13.jpg)
3D Modeling Pipeline
13
Images Sparse Model Dense ModelScene Graph
Sparse Reconstruction
Initialization
Bundle AdjustmentTriangulation
Image Registration Outlier Filtering
Correspondence Search
Feature Extraction
Feature Matching
Geometric Verification
Dense Reconstruction
Multi-View Stereo
Multi-View Fusion
Surface Reconstruction
![Page 14: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/14.jpg)
End-To-End Image-Based 3D Modeling
OpenSource Software: COLMAP https://github.com/colmap/colmap
T. Schöps, J.L. Schönberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, A. Geiger."A Multi-View Stereo Benchmark with ...". CVPR 2017.
A. Knapitsch, J. Park, Q.Y. Zhou, V. Koltun. “Tanks and Temples:Benchmarking Large-Scale Scene Reconstruction”. SIGGRAPH 2017.
14
![Page 15: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/15.jpg)
COLMAP Research Applications
• Papers• Learning the Depths of Moving People by Watching Frozen People, Li et al.,
CVPR 2019, Best Paper Honorable Mention• Neural Rendering in the Wild, Meshry et al., CVPR 2019• Revealing Scenes by Inverting Structure from Motion Reconstructions,
Pittaluga et al., CVPR 2019• Privacy-Preserving Image-Based Localization, Speciale et al., CVPR 2019• Understanding the Limitations of CNN-based Absolute Camera Pose
Regression, Sattler et al., CVPR 2019• D2-Net: A Trainable CNN for Joint Detection and Description of Local
Features by Dusmanu et al., CVPR 2019• …
• Datasets, Workshops, and Benchmarks• Image Matching: Local Features & Beyond, CVPR Workshop 2019• Long-Term Visual Localization under Changing Conditions, CVPR Workshop
2019• MegaDepth Dataset, Li et al., CVPR 2018• Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions,
Sattler et al., CVPR 2018• ETH3D Dataset & Benchmark, Schöps et al., CVPR 2017• …
15
![Page 16: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/16.jpg)
Related Scientific Projects Today
16
![Page 17: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/17.jpg)
Related Scientific Projects Today
17
Sparse Reconstruction
Initialization
Bundle AdjustmentTriangulation
Image Registration Outlier Filtering
Correspondence Search
Feature Extraction
Feature Matching
Geometric Verification
Dense Reconstruction
Multi-View Stereo
Multi-View Fusion
Surface Reconstruction
• Lifelong Mapping Lukas Schmid, ETH Zurich
![Page 18: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/18.jpg)
Related Scientific Projects Today
18
Sparse Reconstruction
Initialization
Bundle AdjustmentTriangulation
Image Registration Outlier Filtering
Correspondence Search
Feature Extraction
Feature Matching
Geometric Verification
Dense Reconstruction
Multi-View Stereo
Multi-View Fusion
Surface Reconstruction
• Learning Feature Representations Mihai Dusmanu, ETH Zurichfor Localization and Mapping
![Page 19: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/19.jpg)
Related Scientific Projects Today
19
Sparse Reconstruction
Initialization
Bundle AdjustmentTriangulation
Image Registration Outlier Filtering
Correspondence Search
Feature Extraction
Feature Matching
Geometric Verification
Dense Reconstruction
Multi-View Stereo
Multi-View Fusion
Surface Reconstruction
• Event Cameras Davide Scaramuzza, University of Zurich
![Page 20: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/20.jpg)
Related Scientific Projects Today
20
Sparse Reconstruction
Initialization
Bundle AdjustmentTriangulation
Image Registration Outlier Filtering
Correspondence Search
Feature Extraction
Feature Matching
Geometric Verification
Dense Reconstruction
Multi-View Stereo
Multi-View Fusion
Surface Reconstruction
• Privacy Preserving Mixed Reality Cloud Pablo Speciale, Microsoft
![Page 21: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/21.jpg)
Related Scientific Projects Today
21
Sparse Reconstruction
Initialization
Bundle AdjustmentTriangulation
Image Registration Outlier Filtering
Correspondence Search
Feature Extraction
Feature Matching
Geometric Verification
Dense Reconstruction
Multi-View Stereo
Multi-View Fusion
Surface Reconstruction
• Adaptive Mesh Texture for Edmond Boyer, INRIAMulti-View Appearance Modeling
![Page 22: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/22.jpg)
Related Scientific Projects Today
22
Sparse Reconstruction
Initialization
Bundle AdjustmentTriangulation
Image Registration Outlier Filtering
Correspondence Search
Feature Extraction
Feature Matching
Geometric Verification
Dense Reconstruction
Multi-View Stereo
Multi-View Fusion
Surface Reconstruction
• Semantic 3D Reconstruction Martin Oswald, ETH Zurich
![Page 23: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/23.jpg)
Other Talks
• Eye Tracking – the Key to Computational Understanding of Human Behavior and CognitionMirko Meboldt, ETH Zurich
• Medical Augmented Reality: State of the Art, Requirements and ChallengesNassir Navab, TU Munich, JHU and Balgrist Hospital
23
![Page 24: (Crowdsourced) Image-based 3D Modeling - microsoft.com · 3D Modeling Pipeline 13 Images Scene Graph Sparse Model Dense Model Sparse Reconstruction Initialization Triangulation Bundle](https://reader036.vdocuments.us/reader036/viewer/2022070704/5e82e38a75ea2b52f6241450/html5/thumbnails/24.jpg)
Questions?
24