super resolution on 3d point clouds using deep learning€¦ · master in computer vision ......
TRANSCRIPT
![Page 1: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/1.jpg)
Super Resolution on 3D Point
Clouds
using Deep Learning
Belén Luque López
![Page 2: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/2.jpg)
About me
Bachelor's Degree in
Audiovisual Systems
Engineering
Master in Computer
Vision
Master’s Thesis
Javier Ruiz Béatrice
Pesquet
Super Resolution on 3D
Point Clouds using Deep
Learning
May - September
2017
2
![Page 3: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/3.jpg)
Motivation
Telepresence project
Internship at the Image Processing Group
of the UPC (September 2016-April 2017)
Capture room
Visualization room
3
![Page 4: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/4.jpg)
Motivation
Low resolution of point clouds!
Possible solutions:
● Use meshes
● Increase the number
of points 4
Telepresence project
![Page 5: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/5.jpg)
Working with 3D
2.5D (RGB-
D)
Multi-view
projection
Voxelizatio
n
Point cloud
Unorganized list of XYZ
coordinates 5
![Page 6: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/6.jpg)
Super resolution
Super resolution in 2D Super resolution in 2.5D
Super resolution on depth map, then construct
point cloud
Upsample image without losing
spatial detail 6
![Page 7: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/7.jpg)
Super resolution
Super resolution in 3D Super resolution in 2D
7
Create more points, fill the holes
![Page 8: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/8.jpg)
Super resolution
Super resolution in 3D Super resolution in 2D
The position of the new points is
already given
Multiple options, not that
easy! 8
![Page 9: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/9.jpg)
C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” CoRR, vol. abs/1612.00593, 2016.
Our first approach
Pointne
t
Novel deep net architecture that directly consumes point clouds
(unordered point sets)
9
![Page 10: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/10.jpg)
Our first approach
The input is an unordered list of XYZ
coordinates!
10
Pointne
t
![Page 11: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/11.jpg)
Our first approach
11
Pointne
t
mini network to learn affine transformation matrix
make the point cloud invariant to rotation/translation
![Page 12: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/12.jpg)
Our first approach
12
Pointne
t
![Page 13: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/13.jpg)
Our first approach
13
Pointne
t X
Y
Z ...
64
XYZ
XYZ
XYZ
n n
![Page 14: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/14.jpg)
Our first approach
14
Pointne
t
![Page 15: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/15.jpg)
Our first approach
15
Pointne
t
10 5 1 3
1 8 5 9
6 2 3 5
9 1 6 3
4 9 3 5
10 9 6 9
![Page 16: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/16.jpg)
Our first approach
16
Pointne
t
![Page 17: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/17.jpg)
Our first approach
Over 6,000 m2 of indoor spaces, almost 700.000.000 points + meshes
Dataset: 2D-3D-Semantics
(http://buildingparser.stanford.edu/dataset.html)
I. Armeni, A. Sax, A. R. Zamir, and S. Savarese,
“Joint 2D-3D-Semantic Data for Indoor Scene
Understanding” ArXiv e-prints, 2017.
17
![Page 18: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/18.jpg)
Our first approach
Input (N
points)
Output (N
points)
Original point cloud (2N
points)
Pointnet
Original point
cloud
50% sampling
Input
data
Ground
truth
Residual training to obtain the double of points
50%
sampling
18
![Page 19: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/19.jpg)
Our first approach
Pointnet
nx
3
Loss:
1. For each output point, find the closest point in the ground truth (nearest neighbor
search with kd-tree)
2. Minimize RMS distance
Output: new XYZ
coordinates
19
![Page 20: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/20.jpg)
Tools
● Data preparation (point cloud downsampling):
PCL library
● Pointnet model: Tensorflow (or third-party
implementation in pyTorch)
○ + Integration with python-pcl for point cloud
processing (nearest neighbor search)
● Data ingestion: h5py library (dataset stored in
HDF5 format)
● Data augmentation (rotation/jittering): numpy
library
● Visualization of results: meshlab software
numpy + scipy 2D
projection
20
![Page 21: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/21.jpg)
Some thoughts
Can we combine the input points (with convolutions, like in 2D images) ?
The input is an unorganized list of 3D points, neighborhoods are not
defined…
→ Semantic segmentation as a previous step?
Multiple solutions exist for the creation of new points. How do we train the
network?
- Minimize distance to closest point in the ground truth
- Minimize distance to the surface (closest face in a mesh)
Do we want the network to create more
points in the less populated areas? How? 21
![Page 22: Super Resolution on 3D Point Clouds using Deep Learning€¦ · Master in Computer Vision ... Javier Ruiz Béatrice Pesquet Super Resolution on 3D Point Clouds using Deep Learning](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec67a2adb0d1917dc626bed/html5/thumbnails/22.jpg)
Thank you!
Belén Luque López