robots that know what they do - davide...
TRANSCRIPT
![Page 1: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/1.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Davide Scaramuzza
University of Zurich
Robotics and Perception Group
http://rpg.ifi.uzh.ch/
![Page 2: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/2.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Scaramuzza, D., Fraundorfer, F., Visual Odometry: Part I - The First 30 Years and Fundamentals, IEEE Robotics and Automation Magazine, Volume 18, issue 4, 2011.
Fraundorfer, F., Scaramuzza, D., Visual Odometry: Part II - Matching, Robustness, and Applications, IEEE Robotics and Automation Magazine, Volume 19, issue 1, 2012.
![Page 3: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/3.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
input output
Image sequence (or video stream)
from one or more cameras attached to a moving vehicle
Camera trajectory (3D structure is a plus):
VO is the process of incrementally estimating the pose of the vehicle by examining the changes that motion induces on the images of its onboard cameras
![Page 4: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/4.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Sufficient illumination in the environment
Dominance of static scene over moving objects
Enough texture to allow apparent motion to be extracted
Sufficient scene overlap between consecutive frames
Is any of these scenes good for VO? Why?
![Page 5: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/5.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Contrary to wheel odometry, VO is not affected by wheel slip in uneven terrain or other adverse conditions.
More accurate trajectory estimates compared to wheel odometry (relative position error 0.1% − 2%)
VO can be used as a complement to
wheel odometry
GPS
inertial measurement units (IMUs)
laser odometry
In GPS-denied environments, such as underwater and aerial, VO has utmost importance
![Page 6: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/6.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/ Image 1 Image 2
![Page 7: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/7.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Image sequence
Feature detection
Feature matching (tracking)
Motion estimation
2D-2D 3D-3D 3D-2D
Local optimization
VO computes the camera path incrementally (pose after pose)
Tk,k-1
Tk+1,k
Ck-1
Ck
Ck+1
SIFT features tracks
![Page 8: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/8.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Ck
Ck+1
Tk,k-1
Tk+1,k
Ck-1
![Page 9: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/9.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
SFM is more general than VO and tackles the problem of 3D reconstruction of both the structure and camera poses from unordered image sets
The final structure and camera poses are typically refined with an offline optimization (i.e., bundle adjustment), whose computation time grows with the number of images
Reconstruction from 3 million images from Flickr.com
Cluster of 250 computers, 24 hours of computation!
Paper: “Building Rome in a Day”, ICCV’09
This video can be seen at
http://youtu.be/kxtQqYLRaSQ
![Page 10: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/10.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
VO is a particular case of SFM
VO focuses on estimating the 3D motion of the camera sequentially (as a new frame arrives) and in real time.
Bundle adjustment can be used (but it’s optional) to refine the local estimate of the trajectory
Terminology: sometimes SFM is used as a synonym of VO
![Page 11: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/11.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Before loop closing After loop closing Image courtesy of Clemente et al. RSS’07
![Page 12: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/12.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
VO only aims to the local consistency of the trajectory
SLAM aims to the global consistency of the trajectory and of the map
VO can be used as a building block of SLAM
VO is SLAM before closing the loop!
The choice between VO and V-SLAM depends on the tradeoff between performance and consistency, and simplicity in implementation.
VO trades off consistency for real-time performance, without the need to keep track of all the previous history of the camera.
Visual odometry
Visual SLAM
Ima
ge
co
urt
esy o
f C
lem
en
te e
t a
l. R
SS
’07
![Page 13: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/13.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Brief history of VO
Problem formulation
Camera modeling and calibration
Motion estimation
Robust estimation
Error propagation
Camera-pose optimization (bundle adjustment)
Discussion
![Page 14: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/14.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
1996: The term VO was coined by Srinivasan to define motion orientation
in honey bees.
1980: First known stereo VO real-time implementation on a robot by Moraveck
PhD thesis (NASA/JPL) for Mars rovers using a sliding camera. Moravec invented a
predecessor of Harris detector, known as Moravec detector
1980 to 2000: The VO research was dominated by NASA/JPL in preparation of
2004 Mars mission (see papers from Matthies, Olson, etc. From JPL)
2004: VO used on a robot on another planet: Mars rovers Spirit and Opportunity
2004. VO was revived in the academic environment
by Nister «Visual Odometry» paper.
The term VO became popular.
![Page 15: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/15.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Brief history of VO
Problem formulation
Camera modeling and calibration
Motion estimation
Robust estimation
Error propagation
Camera-pose optimization (bundle adjustment)
Discussion
![Page 16: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/16.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Tk,k-1
![Page 17: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/17.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Tk,k-1
Tk+1,k
Ck-1
Ck
Ck+1
![Page 18: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/18.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Tk,k-1
Tk+1,k
Ck-1
Ck
Ck+1
![Page 19: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/19.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
...
𝑪𝟎 𝑪𝟏 𝑪𝟑 𝑪𝟒 𝑪𝒏−𝟏 𝑪𝒏
![Page 20: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/20.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Brief history of VO
Problem formulation
Camera modeling and calibration
Motion estimation
Robust estimation
Error propagation
Camera-pose optimization (bundle adjustment)
Discussion
![Page 21: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/21.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
P
F
p' z z'
fzz
1
'
11
f
![Page 22: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/22.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
P
F
p' z z'
fz
1
'
1
f
'zz fz '
![Page 23: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/23.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
P
F
p' z
fz
1
'
1
f
'zz fz '
C
![Page 24: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/24.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
P
F C
Single effective viewpoint
Image plane
![Page 25: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/25.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
u v
p X
Single effective viewpoint
Image plane
![Page 26: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/26.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 27: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/27.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 28: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/28.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Always possible after the camera has been calibrated!
![Page 29: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/29.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
For convenience, points are projected on the unit sphere. Why?
In the perspective case, is it better to use the perspective or the spherical model?
Image courtesy of Micusik & Pajdla, ACCV’04
![Page 30: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/30.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Brief history of VO
Problem formulation
Camera modeling and calibration
Motion estimation
Robust estimation
Error propagation
Camera-pose optimization (bundle adjustment)
Discussion
Image sequence
Feature detection
Feature matching
Motion estimation
2D-2D 3D-3D 3D-2D
Local optimization
![Page 32: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/32.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Makadia et al. «Correspondence-free structure from motion», IJCV’07
![Page 33: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/33.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Global methods are less accurate than feature-based methods and are computationally more expensive.
Feature-based methods require the ability to match (or track) robustly features across frames but are faster and more accurate than global methods. Therefore, most VO implementations are feature based.
Image courtesy of Makadia et al., IJCV’07
![Page 34: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/34.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
![Page 35: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/35.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
Both 𝑓𝑘−1 and 𝑓𝑘 are specified in 2D
The minimal-case solution involves 5-point correspondences
The solution is found by determining the transformation that minimizes the reprojection error of the triangulated points in each image
![Page 36: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/36.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
Both 𝑓𝑘−1 and 𝑓𝑘 are specified in 3D
To do this, it is necessary to triangulate 3D points (e.g. use a stereo camera)
The minimal-case solution involves 3 non-collinear correspondences
The solution is found by determining the aligning transformation that minimizes the 3D-3D distance
![Page 37: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/37.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
Both 𝑓𝑘−1 and 𝑓𝑘 are specified in 3D
To do this, it is necessary to triangulate 3D points (e.g. use a stereo camera)
The minimal-case solution involves 3 non-collinear correspondences
The solution is found by determining the aligning transformation that minimizes the 3D-3D distance
![Page 38: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/38.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
𝑓𝑘−1 is specified in 3D and 𝑓𝑘 in 2D
This problem is known as camera resection or PnP (perspective from n points)
The minimal-case solution involves 3 correspondences
The solution is found by determining the transformation that minimizes the reprojection error
![Page 39: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/39.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
In the monocular case, the 3D structure needs to be triangulated from two adjacent camera views (e.g., 𝐼𝑘−2 and 𝐼𝑘−1) and then matched to 2D image features in a third view (e.g., 𝐼𝑘).
![Page 40: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/40.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
Both 𝑓𝑘−1 and 𝑓𝑘 are specified in 2D
The minimal-case solution involves 5-point correspondences
The solution is found by determining the transformation that minimizes the reprojection error of the triangulated points in each image
![Page 41: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/41.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
![Page 42: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/42.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
![Page 43: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/43.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
1
1
1
1
z
y
x
p
2
2
2
2
z
y
x
p
epipolar plane
0)'( 12 ptpT 0))(( 12 RptpT
0 ][ 1
T
2 pRtp 0 12 pEpT
Rt ][ E essential matrix
Motion estimation
2D-2D 3D-3D 3D-2D
Epipolar constraint
![Page 44: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/44.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
0 12 pEpTEpipolar constraint
Rt ][ E Essential matrix
1
1
1
1
z
y
x
p
2
2
2
2
z
y
x
p Image coordinates on the Unit sphere
![Page 45: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/45.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
which can be solved with SVD
![Page 46: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/46.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
![Page 47: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/47.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Motion estimation
2D-2D 3D-3D 3D-2D
![Page 48: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/48.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Type of correspondences
Monocular Stereo
2D-2D X X
3D-3D X
3D-2D X X
![Page 49: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/49.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Some of the previous motion estimation methods require triangulation of 3D points
Triangulated 3D points are determined by intersecting backprojected rays from 2D image correspondences of at least two image frames
In reality, they never intersect due to
image noise,
camera model and calibration errors,
and feature matching uncertainty
The point at minimal distance from all intersecting rays can be taken as an estimate of the 3D point position
![Page 50: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/50.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
When frames are taken at nearby positions compared to the scene distance, 3D points will exibit large uncertainty
Therefore, 3D-3D motion estimation methods will drift much more quickly than 3D-2D and 2D-2D methods
In fact, the uncertainty introduced by triangulation affects the motion estimation. In fact, in the 3D-to-3D case the 3D position error is minimized, while in the 3D-to-2D and 2D-to-2D cases is the image reprojection error
![Page 51: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/51.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
One way to avoid this consists of skipping frames until the average uncertainty of the 3D points decreases below a certain threshold. The selected frames are called keyframes
Keyframe selection is a very important step in VO and should always be done before updating the motion
. . .
![Page 52: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/52.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
In the Stereo vision case, 3D-2D method exhibits less drift than 3D-3D method
Stereo vision has the advantage over monocular vision that both motion and structure are computed in the absolute scale. It also exhibits less drift.
When the distance to the scene is much larger than the stereo baseline, stereo VO degenerates into monocular VO
Keyframes should be selected carefully to reduce drift
Regardless of the chosen motion computation method, local bundle adjustment (over the last m frames) should be always performed to compute a more accurate estimate of the trajectory. After bundle adjustment, the effects of the motion estimation method are much more alleviated (as long as the initialization is close to the solution)
![Page 53: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/53.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Brief history of VO
Problem formulation
Camera modeling and calibration
Motion estimation
Robust estimation
Error propagation
Camera-pose optimization (bundle adjustment)
Discussion
Image sequence
Feature detection
Feature matching
Motion estimation
2D-2D 3D-3D 3D-2D
Local optimization
![Page 54: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/54.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Matched points are usually contaminated by outliers, that is, wrong data associations
Possible causes of outliers are
image noise,
occlusions,
blur,
changes in view point and illumination for which the mathematical model of the feature detector or descriptor does not account for
For the camera motion to be estimated accurately, outliers must be removed
This is the task of Robust Estimation
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
efirecam-0-0000018959.jpg
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
efirecam-0-0000018959.jpg, INLIERS
100 200 300 400 500 600
50
100
150
200
250
300
350
400
450
![Page 55: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/55.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Error at the loop closure: 6.5 m Error in orientation: 5 deg Trajectory length: 400 m
Before removing the outliers
After removing the outliers
![Page 56: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/56.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 57: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/57.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
• Select sample of 2 points at random
![Page 58: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/58.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
• Select sample of 2 points at random
• Calculate model parameters that fit the data in the sample
![Page 59: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/59.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
• Select sample of 2 points at random
• Calculate model parameters that fit the data in the sample
• Calculate error function for each data point
![Page 60: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/60.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
• Select sample of 2 points at random
• Calculate model parameters that fit the data in the sample
• Calculate error function for each data point
• Select data that support current hypothesis
![Page 61: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/61.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
• Select sample of 2 points at random
• Calculate model parameters that fit the data in the sample
• Calculate error function for each data point
• Select data that support current hypothesis
• Repeat sampling
![Page 62: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/62.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
• Select sample of 2 points at random
• Calculate model parameters that fit the data in the sample
• Calculate error function for each data point
• Select data that support current hypothesis
• Repeat sampling
![Page 63: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/63.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
ALL-INLIER SAMPLE
![Page 64: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/64.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Has been established as the standard method for motion estimation
in the presence of outliers
![Page 65: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/65.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
1. Randomly select a minimal set of point correspondences
2. Compute motion and count inliers
3. Repeat from 1
Has been established as the standard method for motion estimation
in the presence of outliers
![Page 66: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/66.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
1. Randomly select a minimal set of point correspondences
2. Compute motion and count inliers
3. Repeat N times
The number of iterations needed
grows exponentially with the
outliers
~ 1000 iterations!
Has been established as the standard method for motion estimation
in the presence of outliers
![Page 67: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/67.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 68: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/68.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
))1(1log(
)1log(s
pN
![Page 69: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/69.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 70: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/70.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 71: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/71.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
To estimate the motion of a calibrated camera in 6 DoF, we need 5 points [Kruppa, 1913]
Why ?
![Page 72: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/72.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
In 6 DoF we would need 6 points …
… but the scale is unobservable …
… and therefore we only need 6 – 1 = 5 points
[“5-Point RANSAC”, Nister, 2003]
General rule:
Minimum number of points = NDoF - 1
![Page 73: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/73.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
The “5-Point RANSAC” typically needs ~1000 iterations
To reduce the number iterations, we should use a smaller number of points ( < 5 )
Is this possible?
Yes, if we exploit motion constraints!
![Page 74: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/74.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Can we use an even smaller number of points?
Yes, if we exploit the vehicle non-holonomic constraints
For planar motion, only 3 parameters need to be estimated
θ, φ, ρ => 3 DoF
and therefore only 2 points are needed
[“2-Point RANSAC”, Ortin, 2001]
![Page 75: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/75.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Example of Ackerman steering principle Locally circular motion
Wheeled vehicles follow locally circular motion about the Instantaneous Center of Rotation (ICR)
![Page 76: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/76.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Locally circular motion
φ = θ/2 => only 2 parameters (θ, ρ) need to be estimated
and therefore only 1 point is needed
This is the smallest parameterization possible and results in
the most efficient algorithm for removing outliers
D. Scaramuzza. 1-Point-RANSAC Structure from Motion for Vehicle-Mounted Cameras by Exploiting Non-holonomic
Constraints. International Journal of Computer Vision, Volume 95, Issue 1, 2011
D. Scaramuzza. Performance Evaluation of 1-Point-RANSAC Visual Odometry. Journal of Field Robotics, Vol. 28, issue 5, 2011
Example of Ackerman steering principle
Wheeled vehicles follow locally circular motion about the Instantaneous Center of Rotation (ICR)
![Page 77: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/77.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Only 1 iteration
The most efficient algorithm for
removing outliers, up to 800 Hz
Compute θ for
every point
correspondence
1-Point RANSAC is ONLY used to find the inliers.
Motion is then estimated from them in 6DOF
![Page 78: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/78.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
5-Point RANSAC [Nister’03]
2-Point RANSAC [Ortin’01]
1-Point RANSAC [Scaramuzza, IJCV’11, JFR’11]
Number of
iterations
~1000 ~100 1
0 10 20 30 40 50 60 70 80 90 1000
100
200
300
400
500
600
700
800
900
1000
Fraction of outliers in the data (%)
Num
ber
of
itera
tions,
N
Our proposed method
5-point RANSAC
2-point RANSAC
1-point RANSAC
![Page 79: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/79.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
15,000 images collected in Zurich during a over 25 Km path
Image resolution: 640 x 480
![Page 80: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/80.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
This video can be seen at
http://youtu.be/t7uKWZtUjCE
![Page 81: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/81.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 82: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/82.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Is it really better to use minimal sets in RANSAC?
If one is concerned with certain speed requirements, YES
However, might not be a good choice if the image correspondences are very noisy: in this case, the motion estimated from a minimal set wil be inaccurate and will exhibit fewer inliers when tested on all other points
Therefore, when the computational time is not a real concern and one deals with very noisy features, using a non-minimal set may be better than using a minimal set
![Page 83: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/83.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Brief history of VO
Problem formulation
Camera modeling and calibration
Motion estimation
Robust estimation
Error propagation
Camera-pose optimization (bundle adjustment)
Discussion
Image sequence
Feature detection
Feature matching
Motion estimation
2D-2D 3D-3D 3D-2D
Local optimization
![Page 84: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/84.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Ck
Ck+1
Tk
Tk+1
Ck-1
The uncertainty of the camera pose 𝐶𝑘 is a combination of the uncertainty at 𝐶𝑘−1 (black-solid ellipse) and the uncertainty of the transformation 𝑇𝑘 (gray dashed ellipse)
𝐶𝑘 = 𝑓(𝐶𝑘−1, 𝑇𝑘)
The combined covariance ∑𝑘is
The camera-pose uncertainty is always increasing when concatenating transformations. Thus, it is important to keep the uncertainties of the individual transformations small
![Page 85: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/85.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Brief history of VO
Problem formulation
Camera modeling and calibration
Motion estimation
Robust estimation
Error propagation
Camera-pose optimization (bundle adjustment)
Discussion
Image sequence
Feature detection
Feature matching
Motion estimation
2D-2D 3D-3D 3D-2D
Local optimization
![Page 86: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/86.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
So far we assumed that the transformations are between consecutive frames
Transformations can be computed also between non-adjacent frames 𝑇𝑒𝑖𝑗
and can be used as additional constraints to improve cameras
poses by minimizing the following
For efficiency, only the last 𝑚 keyframes are used
Levenberg-Marquadt can be used
...
𝒎
𝑻𝟑,𝟏 𝑻𝟒,𝟏 𝑻𝒏−𝟏,𝟑
𝑪𝒏−𝒎 𝑪𝒏−𝒎+𝟏 𝑪𝒏−𝒎+𝟐 𝑪𝒏−𝒎+𝟑 𝑪𝒏−𝟏 𝑪𝒏
𝑻𝟏 𝑻𝟐 𝑻𝟑 𝑻𝒏 ...
![Page 87: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/87.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Similar to pose-optimization but it also optimizes 3D points
In order to not get stuck in local minima, the initialization should be close the minimum
Levenberg-Marquadt can be used
...
𝒎
𝑻𝟑,𝟏 𝑻𝟒,𝟏 𝑻𝒏−𝟏,𝟑
𝑻𝟏 𝑻𝟐 𝑻𝟑 𝑻𝒏
𝑪𝒏−𝒎 𝑪𝒏−𝒎+𝟏 𝑪𝒏−𝒎+𝟐 𝑪𝒏−𝒎+𝟑 𝑪𝒏−𝟏 𝑪𝒏
...
![Page 88: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/88.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Loop constraints are very valuable constraints for pose graph optimization
These constraints form graph edges between nodes that are usually far apart and between which large drift might have been accumulated.
Events like reobserving a landmark after not seeing it for a long time or coming back to a previously-mapped area are called loop detections
Loop constraints can be found by evaluating visual similarity between the current camera images and past camera images.
Visual similarity can be computed using global image descriptors or local image descriptors (see lecture about Visual SLAM)
First observation
Second observation after a loop
Image courtesy of Cummins & Newman, IJRR’08
![Page 89: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/89.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Windowed BA reduces the drift compared to 2-view VO because incorporates constraints between several frames
More precise than camera-pose optimization
The choise of the window size m is governed by computational reasons
The computational complexity of BA is 𝑂 𝑞𝑁 + 𝑙𝑚 3 with 𝑁 being
the number of points, 𝑚 the number of poses, and 𝑞 and 𝑚 the number of parameters for points and camera poses
![Page 90: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/90.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
Other sensors can be used such as
IMU (called inertial VO)
Compass
GPS
Laser
An IMU combined with a single camera allows the estimation of the absolute scale. Why?
Make sure that you have many points (thoudsands) which cover the image uniformly
![Page 91: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/91.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
VO has successfully been applied within various technological fields
Space exploration:
Planetary lander furing descent phase
Spirit and Opportunity Mars-exploration rovers
Since 2004, used VO in addition to dead-reckoning for about 6 Km
Especially in preence of wheel slip
MAV navigation
European project SFLY
Vision-based MAVs at the Robotics and Perception Group (see http://rpg.ifi.uzh.ch/research_mav.html )
Underwater vehicles
Automotive industry
The sFly video can be seen at
http://youtu.be/_-p08o_oTO4
![Page 92: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/92.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
World-first mouse scanner
Currently distributed by LG: SmartScan LG LSM100
This video can be seen at
http://youtu.be/A4NGXFv27AE
![Page 93: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/93.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/
![Page 94: Robots that know what they do - Davide Scaramuzzarpg.ifi.uzh.ch/docs/Visual_Odometry_Tutorial.pdf · 2013-11-18 · Visual odometry VO is SLAM before closing the loop! The choice](https://reader030.vdocuments.us/reader030/viewer/2022040210/5e55ac0183114d5f2804ac7a/html5/thumbnails/94.jpg)
Copyright of Davide Scaramuzza - [email protected] - https://sites.google.com/site/scarabotix/