sm3d: real-time simultaneous mapping and 3d...
TRANSCRIPT
![Page 1: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/1.jpg)
SM3D: Real-Time Simultaneous Mapping and 3D Detection
Group Member: Guanqing Li, Rui Yang, Runfa Li, Shuangquan Sun
Mentors: Prof. Trivedi, Akshay Rangesh
Presentation Date: 06/02/2020
![Page 2: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/2.jpg)
• Structure from Motion Module - Unsupervised• 3D Detection Module – Supervised
Input: RGB Images: 𝐼!, 𝐼!"#, 𝐼!"$Depth Images: 𝐷!"#GPS Location: 𝑃!"#, 𝑃!"$
Output:Mapping: 𝑀𝑎𝑝% 𝑡𝑜 𝑀𝑎𝑝!Depth Images: 𝐷!3D Bounding Box: 𝐵𝑜𝑥! 2
METHODOLOGY
![Page 3: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/3.jpg)
• Expected goal
Ø Real-Time simultaneous mapping & 3D detection
• Novelty
Ø First time combining Structure from Motion (SFM) with 3Ddetection using Pseudo Lidar point cloud
• Significance
Ø A real-time system with diverse information (Mapping, 3Dbounding box, depth image, Pseudo Lidar point cloud) whichleaves slots for many further modification and improvement
3
EXPECTED GOALS, NOVELTY & SIGNIFICANCE
![Page 4: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/4.jpg)
SIFT SuperPoint network Libviso2 built-in function
Original SFM learner [1] Advanced SFM learner
• KITTI sequence09: 0-700 frame
• Blue line: estimated trajectory
• Red line: GT trajectroy
4
CURRENT PROGRESS: SFM
![Page 5: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/5.jpg)
CURRENT PROGRESS: SFM
• Complete KITTI sequence 09: 1590 frames
• Blue line: estimated trajectory
• Red line: GT trajectroy
Original SFM Learner [1] Advanced SFM Learner(this work)
5
![Page 6: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/6.jpg)
• ATE: Absolute trajectory error
• RE: Rotation Error
• Advanced VS Original
REadv ≈ REori
ATEadv < ATEori
ATE RE
Original 0.0100 0.0016
Advanced 0.0091 0.0017
KITTI 09
KITTI 10
ATE RE
Original 0.0085 0.0016
Advanced 0.0081 0.0018
6
CURRENT PROGRESS: SFM
![Page 7: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/7.jpg)
7
CURRENT PROGRESS: PSEUDO-LIDAR POINT CLOUD
Image Ground Truth Depth Map
Pseudo-LiDAR Point Cloud
Predicted Depth Map (Need improvement)
![Page 8: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/8.jpg)
8
CURRENT PROGRESS: 3D DETECTION
![Page 9: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/9.jpg)
9
It-1I2I It
Vt-1 VtV2V
V2I I2V
Image:
Velodyne:
• V2I : Velodyne → Reference → Rectified → Camera → Image(computationally expensive)
• V2V = I2I(If no relative motion between Lidar & Camera)
3D 3D 3D 3D 2D
CURRENT PROGRESS: POINT STACKING ALGORITHM
![Page 10: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/10.jpg)
10
Stack Number = 3
• Keep the total point number same after “Stack Number” by removing the earliest points
time: t = 0 t = 4t = 3t = 2t = 1 t = T
V0 V1 VTV2 V3 V4Velodyne: …Net point number: n0 n1 n2 n3 n4 nT
Total point number: n0 N1 = n0+n1 N = N1 +n2 N =(N-n3)+n3 N =(N-n3)+n3 N =(N-nT)+nT
CURRENT PROGRESS: POINT STACKING ALGORITHM
![Page 11: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/11.jpg)
CURRENT PROGRESS: POINT STACKING RESULT
11
Pseudo-LiDAR Point Cloud before Augmentation Pseudo-LiDAR Point Cloud after Augmentation(Stack number = 2)
![Page 12: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/12.jpg)
CURRENT PROGRESS: REMAINING ISSUES
12
• Improve the quality of predicted depth mapsØ Choose appropriate hyperparameters to avoid overfitting of disparity lossØ Now are using GT depth for following stepØ Quality of the depth image decide the performance of Lidar & 3D detection
• Train Pointpillar, include 3D detection into the architecture while maintain efficiency.Ø Mapping alone 68 ms/frame on week 6,
182 ms/frame this week (same code, same environment, server busy)
Disparity Loss Pose Loss Total Loss
![Page 13: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/13.jpg)
INDIVIDUAL WORKLOAD
13
Ø Depth map and projection matrix: Runfa
Ø Depth to pseudo Lidar point clouds: All
Ø 3D detection on pseudo lidar point clouds
PointPillars (high speed): All
Debugging and documentation: Shuangquan
Ø Stacking Pseudo-LiDAR Point Clouds: Runfa
![Page 14: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/14.jpg)
MILESTONES & TIMELINE
14
• Week 10:
Ø 3D detection
• Week 11:
Ø Report Writing
Ø Add supervision to 3D detection, end-to-end multi-task learning (flexible)
![Page 15: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/15.jpg)
REFERENCE
[1] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and
ego-motion from video. In CVPR, 2017. 1, 3, 6, 7, 8
[2] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger.
Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for
autonomous driving. In CVPR, 2019.
[3] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar
Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR,
2019. 1, 2, 6, 7, 8, 14, 15, 16
15
![Page 16: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/16.jpg)
THANKS
![Page 17: SM3D: Real-Time Simultaneous Mapping and 3D Detectioncvrr.ucsd.edu/ece285sp20/files/team1-final.pdf · 2020. 6. 3. · SM3D: Real-Time Simultaneous Mapping and 3D Detection Group](https://reader036.vdocuments.us/reader036/viewer/2022081620/6107f887bd34d103830e78b8/html5/thumbnails/17.jpg)
QUESTIONS?