surface normal prediction using hypercolumn skip-net & normal-depth
TRANSCRIPT
![Page 1: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/1.jpg)
Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth conversion by
Poisson Equation Ching-Hang Chen
Cheng-An Hou
![Page 2: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/2.jpg)
Introduction• 3D properties by 2D RGB images
• 3D scene understanding• Object geometry• Object location (near/far) • Segmentation
• Leverage Deep Learning to solve 3D from monocular images• Surface normal• Depth
![Page 3: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/3.jpg)
Related Work• Traditional approaches• Machine learning, deep learning
![Page 4: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/4.jpg)
Traditional Approach• Structure from motion(SfM)
Imaging Feature Point Matching
Solve Fundamental Matrix
SceneShape
by Epi-polar geometry
![Page 5: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/5.jpg)
Traditional Approach• Photometric Stereo
Imaging Surface Normal Estimation
Surface Normal Integration
Scene Shape
…
𝑵=𝑰 𝑳−1
Normal field
![Page 6: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/6.jpg)
Limitation of Traditional Methods• Need Calibration: camera or light source• In general, assume Lambertian surface• The imaging step is usually impractical
![Page 7: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/7.jpg)
Depth from 2D Images: Challenges • Ambiguity from 2D to 3D, and size• Complex light properties:
• Reflection• Refraction• Inter-Reflection• Sub-surface Scattering
![Page 8: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/8.jpg)
Reflection
![Page 9: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/9.jpg)
Refraction
![Page 10: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/10.jpg)
Inter-reflections
![Page 11: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/11.jpg)
Subsurface Scattering
![Page 12: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/12.jpg)
• How can machine learning help?• How human infer the 3D properties from 2D images?
![Page 13: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/13.jpg)
Machine Learning Approach
Fouhey, David F., Abhinav Gupta, and Martial Hebert. "Data-driven 3D primitives for single image understanding." Proceedings of the IEEE International Conference on Computer Vision. 2013.
![Page 14: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/14.jpg)
Deep Learning Approach
Wang, Xiaolong, David Fouhey, and Abhinav Gupta. "Designing deep networks for surface normal estimation." CVPR 2015.
Bansal, Aayush and Russell, Bryan and Gupta, Abhinav. "Marr Revisited: 2D-3D Model Alignment via Surface Normal Prediction. " CVPR 2016.
Liu, Fayao, Chunhua Shen, and Guosheng Lin. "Deep convolutional neural fields for depth estimation from a single image." CVPR 2015
Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV2015.
![Page 15: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/15.jpg)
Deep Learning Approach
Liu, Fayao, Chunhua Shen, and Guosheng Lin. "Deep convolutional neural fields for depth estimation from a single image." CVPR 2015
![Page 16: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/16.jpg)
Deep Learning Approach
Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV2015.
![Page 17: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/17.jpg)
Problem Formulation• Pixel-wise predict surface normal for monocular images by CNN
features• Explore the relation between surface normal and depth, conversion
between the two
![Page 18: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/18.jpg)
Dataset: NYU v2• Indoor Scenes collected by Microsoft Kinect, 407,024 frames• RGB, Depth, Surface Normal maps
• Why predict Surface Normal instead of Depth?
Silberman, Nathan, et al. "Indoor segmentation and support inference from RGBD images." Computer Vision–ECCV 2012.
![Page 19: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/19.jpg)
Proposed Network
… … … …
NN Upsampling
…
Pretrained VGG
Convolutional Layers
128 x 128
64 x 64
Hypercolumn
128 256 512 512
1408
64
4096 40
![Page 20: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/20.jpg)
Training• Classification instead of Regression• 40 classes of surface normal• Loss: negative log likelihood• Optimization: SGD
![Page 21: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/21.jpg)
Conv1 layer Visualization (AlexNet)
![Page 22: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/22.jpg)
Experiment Result: Surface Normal• Quantitative results:
Mean Median 11.25 22.50 30.00
AlexNet (Pretrain) 29.2 24.6 22.5 46.0 58.7
AlexNet (Scratch) 29.6 24.2 23.6 46.9 58.8
Ladicky et al. [3] 35.5 25.5 24.0 45.6 55.9
Wang, et al. [2] 28.8 17.9 35.2 57.1 65.5
Eigen, et al. [1] 25.9 18.2 33.2 57.5 67.7
Skip-Net (ours) 24.7 17.7 35.6 58.6 68.0
[1] Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV 2015.
[2] Wang, Xiaolong, David Fouhey, and Abhinav Gupta. "Designing deep networks for surface normal estimation." CVPR 2015.
[3] Zeisl, Bernhard, and Marc Pollefeys. "Discriminatively trained dense surface normal estimation." Computer Vision–ECCV 2014.
![Page 23: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/23.jpg)
Experiment Result: Surface Normal
RGB image
Surface Normal
![Page 24: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/24.jpg)
Depth Map & Surface Normal
• Theoretically, Depth Map can be differentiated to derive Surface Normal, and Surface Normal can be integrated to obtain Depth Map.
• Depth Map has ambiguity of scale, and Surface Normal does not.
Depth Map Surface Normal
• NYU v2 Surface Normal Acquisition
Silberman, Nathan, et al. "Indoor segmentation and support inference from RGBD images." Computer Vision–ECCV 2012.
![Page 25: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/25.jpg)
Depth from Surface Normal by Orthogonality
![Page 26: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/26.jpg)
Depth from Surface Normal by Poisson Equation
• Minimize the objective J(v), where v is the depth function v(x,y)
M. Breuß, Y. Qu´eau, M. B¨ahr, and J.-D. Durou. Highly efficient surface normal integration. In Proceedings of the Conference Algoritmy, pages 204–213, 2016
![Page 27: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/27.jpg)
Experiment Result: ComparisonOrthogonality Poisson
![Page 28: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/28.jpg)
Experiment Result: Depth Map from Surface Normal
![Page 29: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/29.jpg)
Experiment Result: Depth Map from Surface Normal
![Page 30: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/30.jpg)
More Experiment Examples
![Page 31: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/31.jpg)
Discussion • To solve task such as surface normal prediction from monocular
images, information insufficiency could be resolved by learning from dataset
• Limitation of depth conversion from surface normal: discontinuous boundaries, and wrong surface normal prediction
• The objective function for predicting depth is to minimize the overall depth prediction in the scene, object’s local structure might not be discovered
![Page 32: Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth](https://reader035.vdocuments.us/reader035/viewer/2022062902/58edb9161a28ab2b638b46d5/html5/thumbnails/32.jpg)
•Thank you for your attention!