stereo visual odometry - vmlblog.free.frvmlblog.free.fr/projects/slam/diaporama-compressed.pdf · 2...
TRANSCRIPT
SALMA
Stereo Visual Odometry
April 26 2019
Victor MARTIN-LAC
2
Features
Hardware
● Build a stereo rig
Software
● user interface allowing to
– record multiple view video stream,
– calibrate cameras and stereo rigs,
– compute visual odometry trajectory,
– display trajectory and odometry data.
● stereo visual odometry algorithm.
● algorithm to track calibration target.
● multiple view video stream acquisition.
3
Numbers
Begun in august 2018
About 20 000 lines of C++
About 1100 € spent
http://github.com/vmartinlac/salma/
Hardware
5
USB
Ethernet * 2I/O * 2
Rig
Arduino PC
Architecture
6
Architecture
Take Received Image
Trigger
Process Image
Trigger
7
Cameras and Lenses
2* Allied Vision TechnologiesManta G125C
Gigabit Ethernet1292 * 964 @ 30 fps
2 * Fujinon TF8DA-8B8mm F/2.2 3-CCD Lens
FoV 33° × 25°
2 * KOWA LM4NCL3.5mm F1.4 – 16FoV 76.7° x 57.7°
8
The Platform
● Designed in FreeCAD● 3d-printed at http://www.fabric-insa.fr/
9
Power and Trigger
10
PC
My PC 500 Go SSD drive Gigabit ethernetexpresscard
11
Other Stuff
CAT7 RJ45 cable * 2 I/O cable with Hirose12-pin connector * 2
12V 3A DCpower supply
13
Putting It All Together
PC CameraCalibrationTarget
Stereo Rig
Arduino
RigCalibration
Target
Hardware Legacy
15
The Platform (legacy)
23 cm baseline → hard to produce dense disparity maps in room-scale environment
16
Ethernet * 2I/O * 2
Rig
Signal Gen. PC
Architecture (legacy)
Hard to associate frames received from several cameras
17
Power and Trigger (legacy)
24 Hz square wave generator
18
Calibration Targets (legacy)
Check that background of SVG file is white, not transparent!
Software
20
Dependencies and ToolsDependencies
● Aravis
● Eigen
● g2o
● libjpeg-turbo
● Nanoflann
● OpenCV
● Openscenegraph
● Qt
● Sophus
Tools:
● C++
● Cmake
Portability: Linux only!
● Aravis only on Linux
● Use of open() stat() write() close()
21
Software Architecture
slam
persistence
ui
targetmvpnpbuildinfo videocalibration stereomatcher
uicommon
uirecording uicalibration uireconstruction
Software
Video Acquisition & Recording
23
Video Acquisition
● Rely on Aravis library as implementation of GenICam norm (Vimba SDK from Allied Vision Technologies can be used from only one user thread)
● Some multithreading involved.
● Minimize CPU usage.
24
Video Recording
● Use libjpeg-turbo for encoding
● Synchronous write on file system
– avoid sudden increase of write time when file system cache gets full.
Software
Detecting the Calibration Target
26
Detecting the calibration targetAlgorithmic steps:
1) Good Features to Track
2) Thresholding
3) Interest point filtering
4) Computation of topology and edge filtering
5) Propagation of a coordinate system
6) Dots detection and correction of orientation
7) Numbering the points
27
Detecting the calibration target
(1) (2) (3)
28
Detecting the calibration target
(4) (5) (6)
29
Detecting the calibration target
(7) (8) (9)
30
Detecting the calibration target
Implemented and potential applications of this algorithm:
● Camera calibration (extrinsic and/or intrinsic parameters)
● Rig calibration (relative pose of cameras)
● Pose estimation
Software
Stereo Visual Odometry Method
32
Why Stereo?Pro stereo:
● Easier initialization
● Scale does not drift
● Handle pure rotation
Pro mono:
● Half cost
● Half amount of data
33
Algorithm
Triangulation
Stereo Matching
Local Bundle Adjustment
Keyframe Selection
Alignment
Temporal Matching
Feature Extraction
● Sparse indirect method
● Same family as PTAM and ORB-SLAM.
34
Feature Extraction● Purpose: Detect and describe key points.
● Use ORB features.
35
Temporal Matching● Purpose: establish 2d-3d correspondences between keypoints and local map points.
● Difference with ORB-SLAM: no projection.
● Match left (right) view with k previous left (right) views.
36
Alignment● Purpose: compute pose of stereo rig given 2d-3d correspondances.
● Multiple-view perspective-N-points (MVPnP).
● Use Levenberg-Marquardt + RANSAC.
37
Keyframe Selection● Purpose: decide whether this frame is to be regarded as keyframe or not.
● Check whether translation or rotation angle since last keyframe are above some threshold.
38
Local Bundle Adjustment● Purpose: adjust poses of last n keyframes and positions of visible mappoints to
minimize reprojection error.
● Multiple view.
39
Stereo Matching● Purpose: match keypoints from left and right views.
● Criteria: epipolar constraints, Lowe criterion, distance between descriptors.
40
Triangulation● Purpose: from each stereo match, compute corresponding map point.
● Correction of key point positions according to epipolar constraints (Lindstrom method).
● Middle of common perpendicular.
41
Results
42
Results
43
Runtime● Minimizing runtime is not the focus of the project.
● Pipeline can be CUDA-accelerated if compiled with some CMake option.
● On my Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
– FEATURES DETECTION: 472 ms
– TEMPORAL MATCHER: 50 ms
– ALIGNMENT: 24 ms
– KEYFRAME SELECTION: 0 ms
– LOCAL BUNDLE ADJUSTMENT: 418 ms
– STEREO MATCHING: 52 ms
– TRIANGULATION: 0 ms
Software
User Friendlinesses
45
Data Persistence
For each project:
● Images are stored individually on the file system as JPEG.
● Everything else is stored in one main SQLITE database:
● Recording metadata,
● Calibration data,
● Reconstruction data.
46
User Interface and 3d Visualization● Qt5 for user interface.
● Openscenegraph for 3d visualization.
Conclusion
48
● Evaluate odometry precision and robustness
● Implement dense reconstruction (nice to have)
– Experimented with libelas and OpenCV
– Read some papers
What Next?
49
Thank you for your attention
Questions & Answers