CoReNet: Coherent 3D scene reconstruction from a single RGB image

ECCV 2020 OralPresenter: Yu Han

STRUCT Group Seminar2020/9/15

1

Outline• Authors• Background• Network• Experiments & Results• Conclusion

2

3

3d-r2n2: A unified approach for single and multi-view 3d object reconstruction, ECCV, 2016

Backgroundp 3D-R2N2

p Encoder: 2D CNN

p Decoder: 3D LSTM

4

Pixel2mesh: Generating 3d mesh models from single RGB images, ECCV, 2018

Backgroundp Pix2Mesh

5

Occupancy networks: Learning 3d reconstruction in function space, CVPR, 2019

Backgroundp ONN

Coherent 3D Scene ReconstructionMain Contributions

p Ray traced skip connections.p Hybrid volume representation. p IoU loss.

6

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Networksp Encoder-decoder architecture

7

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

3D Volume Representationp Voxel grid representation à probability distribution over C possible classesp Arbitrary fine discretization of the volume: 1/n output resolutionp Repeat

8

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Ray Traced Skip Connectionsp Decoder points à encoder image àsample encoding image à 3D decoderp Rely on known camera parametersp Reconstructing relative to camera

9

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

IoU Training Lossp C possible classes (including void)p Categorical cross-entropy + focal loss à individual lossesp Overall object geometry

10

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Experimentsp Dataset

p ShapeNet: synthetic imagesp Pix3D: real image

11

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Experimentsp Ablation Study

12

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Experimentsp Comparison to state-of-the-art

13

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Experimentsp Single object reconstruction

14

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Experimentsp Single object reconstruction

15

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Experimentsp Multiple object reconstruction

16

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

ConclusionpRay traced skip connections.p Hybrid volume representation. p IoU loss.p Coherent reconstruction.

17

CoReNet: Coherent 3D scene reconstruction from a single RGB image, ECCV, 2020

Thank You

18