surface normal prediction using hypercolumn skip-net & normal-depth

Surface Normal Prediction using Hypercolumn Skip-Net & Normal-Depth conversion by

Poisson Equation Ching-Hang Chen

Cheng-An Hou

Introduction• 3D properties by 2D RGB images

• 3D scene understanding• Object geometry• Object location (near/far) • Segmentation

• Leverage Deep Learning to solve 3D from monocular images• Surface normal• Depth

Related Work• Traditional approaches• Machine learning, deep learning

Traditional Approach• Structure from motion(SfM)

Imaging Feature Point Matching

Solve Fundamental Matrix

SceneShape

by Epi-polar geometry

Traditional Approach• Photometric Stereo

Imaging Surface Normal Estimation

Surface Normal Integration

Scene Shape

…

𝑵=𝑰 𝑳−1

Normal field

Limitation of Traditional Methods• Need Calibration: camera or light source• In general, assume Lambertian surface• The imaging step is usually impractical

Depth from 2D Images: Challenges • Ambiguity from 2D to 3D, and size• Complex light properties:

• Reflection• Refraction• Inter-Reflection• Sub-surface Scattering

Reflection

Refraction

Inter-reflections

Subsurface Scattering

• How can machine learning help?• How human infer the 3D properties from 2D images?

Machine Learning Approach

Fouhey, David F., Abhinav Gupta, and Martial Hebert. "Data-driven 3D primitives for single image understanding." Proceedings of the IEEE International Conference on Computer Vision. 2013.

Deep Learning Approach

Wang, Xiaolong, David Fouhey, and Abhinav Gupta. "Designing deep networks for surface normal estimation." CVPR 2015.

Bansal, Aayush and Russell, Bryan and Gupta, Abhinav. "Marr Revisited: 2D-3D Model Alignment via Surface Normal Prediction. "  CVPR 2016.

Liu, Fayao, Chunhua Shen, and Guosheng Lin. "Deep convolutional neural fields for depth estimation from a single image." CVPR 2015

Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV2015.

Deep Learning Approach

Liu, Fayao, Chunhua Shen, and Guosheng Lin. "Deep convolutional neural fields for depth estimation from a single image." CVPR 2015

Deep Learning Approach

Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV2015.

Problem Formulation• Pixel-wise predict surface normal for monocular images by CNN

features• Explore the relation between surface normal and depth, conversion

between the two

Dataset: NYU v2• Indoor Scenes collected by Microsoft Kinect, 407,024 frames• RGB, Depth, Surface Normal maps

• Why predict Surface Normal instead of Depth?

Silberman, Nathan, et al. "Indoor segmentation and support inference from RGBD images." Computer Vision–ECCV 2012.

Proposed Network

… … … …

NN Upsampling

…

Pretrained VGG

Convolutional Layers

128 x 128

64 x 64

Hypercolumn

128 256 512 512

1408

64

4096 40

Training• Classification instead of Regression• 40 classes of surface normal• Loss: negative log likelihood• Optimization: SGD

Conv1 layer Visualization (AlexNet)

Experiment Result: Surface Normal• Quantitative results:

Mean Median 11.25 22.50 30.00

AlexNet (Pretrain) 29.2 24.6 22.5 46.0 58.7

AlexNet (Scratch) 29.6 24.2 23.6 46.9 58.8

Ladicky et al. [3] 35.5 25.5 24.0 45.6 55.9

Wang, et al. [2] 28.8 17.9 35.2 57.1 65.5

Eigen, et al. [1] 25.9 18.2 33.2 57.5 67.7

Skip-Net (ours) 24.7 17.7 35.6 58.6 68.0

[1] Eigen, David, and Rob Fergus. "Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.“ ICCV 2015.

[2] Wang, Xiaolong, David Fouhey, and Abhinav Gupta. "Designing deep networks for surface normal estimation." CVPR 2015.

[3] Zeisl, Bernhard, and Marc Pollefeys. "Discriminatively trained dense surface normal estimation." Computer Vision–ECCV 2014.

Experiment Result: Surface Normal

RGB image

Surface Normal

Depth Map & Surface Normal

• Theoretically, Depth Map can be differentiated to derive Surface Normal, and Surface Normal can be integrated to obtain Depth Map.

• Depth Map has ambiguity of scale, and Surface Normal does not.

Depth Map Surface Normal

• NYU v2 Surface Normal Acquisition

Silberman, Nathan, et al. "Indoor segmentation and support inference from RGBD images." Computer Vision–ECCV 2012.

Depth from Surface Normal by Orthogonality

Depth from Surface Normal by Poisson Equation

• Minimize the objective J(v), where v is the depth function v(x,y)

M. Breuß, Y. Qu´eau, M. B¨ahr, and J.-D. Durou. Highly efficient surface normal integration. In Proceedings of the Conference Algoritmy, pages 204–213, 2016

Experiment Result: ComparisonOrthogonality Poisson

Experiment Result: Depth Map from Surface Normal

Experiment Result: Depth Map from Surface Normal

More Experiment Examples

Discussion • To solve task such as surface normal prediction from monocular

images, information insufficiency could be resolved by learning from dataset

• Limitation of depth conversion from surface normal: discontinuous boundaries, and wrong surface normal prediction

• The objective function for predicting depth is to minimize the overall depth prediction in the scene, object’s local structure might not be discovered

•Thank you for your attention!