![Page 1: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/1.jpg)
Pose Estimation16-385 Computer Vision (Kris Kitani)
Carnegie Mellon University
![Page 2: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/2.jpg)
Structure(scene geometry)
Motion (camera geometry) Measurements
Pose Estimation known estimate 3D to 2D correspondences
Triangulation estimate known 2D to 2D coorespondences
Reconstruction estimate estimate 2D to 2D coorespondences
![Page 3: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/3.jpg)
Given a single image, estimate the exact position of the photographer
Pose Estimation
![Page 4: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/4.jpg)
Pose estimation for digital display
![Page 5: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/5.jpg)
Given a set of matched points
and camera model
Find the (pose) estimate of
parametersprojection model
point in 3D space
point in the image
3D Pose Estimation(Resectioning, Calibration, Perspective n-Point)
{Xi,xi}
x = f(X;p) = PXCamera matrix
P
![Page 6: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/6.jpg)
Recall: Camera Models (projections)
Weak perspective
Perspective
X
�
Z̄fC
Parallel (Orthographic)
world point
We’ll use a perspective camera model for pose estimation
![Page 7: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/7.jpg)
What is Pose Estimation?
X world point
Ocamera
Oimage
Oworld
Given
![Page 8: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/8.jpg)
X world point
f = ?
Ocamera
Oimage
py = ?
R = ?
t = ?
Oworld
Estimate
What is Pose Estimation?
extrinsic parameters
intrinsic parameters
![Page 9: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/9.jpg)
Same setup as homography estimation using DLT (slightly different derivation here)
![Page 10: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/10.jpg)
2
4x
y
z
3
5 =
2
4p1 p2 p3 p4
p5 p6 p7 p8
p9 p10 p11 p12
3
5
2
664
X
Y
Z
1
3
775
x
0 =p>1 X
p>3 X
y0 =p>2 X
p>3 X
Inhomogeneous coordinates
(non-linear correlation between coordinates)
Mapping between 3D point and image points
How can we make these relations linear?
2
4x
y
z
3
5 =
2
4—– p>
1 —–—– p>
2 —–—– p>
3 —–
3
5
2
4 X
3
5
What are the unknowns?
![Page 11: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/11.jpg)
x
0 =p>1 X
p>3 X
y0 =p>2 X
p>3 X
p>2 X � p>
3 Xy0 = 0
p>1 X � p>
3 Xx
0 = 0
Make them linear with algebraic manipulation…
Now you can setup a system of linear equations with multiple point correspondences
(this is just DLT for different dimensions)
How can we make these relations linear?
![Page 12: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/12.jpg)
p>2 X � p>
3 Xy0 = 0
p>1 X � p>
3 Xx
0 = 0
In matrix form …
X> 0 �x
0X>
0 X> �y
0X>
�2
4p1
p2
p3
3
5 = 0
For N points … 2
666664
X>1 0 �x
0X>1
0 X>1 �y
0X>1
......
...X>
N 0 �x
0X>N
0 X>N �y
0X>N
3
777775
2
4p1
p2
p3
3
5 = 0
![Page 13: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/13.jpg)
Solve for camera matrix by
ˆ
x = argmin
x
kAxk2 subject to kxk2 = 1
A =
2
666664
X>1 0 �x
0X>1
0 X>1 �y
0X>1
......
...X>
N 0 �x
0X>N
0 X>N �y
0X>N
3
777775x =
2
4p1
p2
p3
3
5
SVD!
![Page 14: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/14.jpg)
Solve for camera matrix by
ˆ
x = argmin
x
kAxk2 subject to kxk2 = 1
A =
2
666664
X>1 0 �x
0X>1
0 X>1 �y
0X>1
......
...X>
N 0 �x
0X>N
0 X>N �y
0X>N
3
777775x =
2
4p1
p2
p3
3
5
A = U⌃V>
=9X
i=1
�iuiv>i
Solution x is the column of V corresponding to smallest singular
value of
![Page 15: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/15.jpg)
Solve for camera matrix by
ˆ
x = argmin
x
kAxk2 subject to kxk2 = 1
A =
2
666664
X>1 0 �x
0X>1
0 X>1 �y
0X>1
......
...X>
N 0 �x
0X>N
0 X>N �y
0X>N
3
777775x =
2
4p1
p2
p3
3
5
Equivalently, solution x is the Eigenvector corresponding to
smallest Eigenvalue of A>A
![Page 16: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/16.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
How do you get the intrinsic and extrinsic parameters from the projection matrix?
Almost there …
![Page 17: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/17.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
Decomposition of the Camera Matrix
![Page 18: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/18.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
P = K[R|t]= K[R|�Rc]
= [M|�Mc]
Decomposition of the Camera Matrix
![Page 19: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/19.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
P = K[R|t]= K[R|�Rc]
= [M|�Mc]
Decomposition of the Camera Matrix
![Page 20: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/20.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
P = K[R|t]= K[R|�Rc]
= [M|�Mc]
Decomposition of the Camera Matrix
Find the camera center C Find intrinsic K and rotation R
![Page 21: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/21.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
P = K[R|t]= K[R|�Rc]
= [M|�Mc]
Decomposition of the Camera Matrix
Find the camera center C Find intrinsic K and rotation R
Pc = 0
![Page 22: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/22.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
P = K[R|t]= K[R|�Rc]
= [M|�Mc]
SVD of P!c is the Eigenvector corresponding to
smallest Eigenvalue
Decomposition of the Camera Matrix
Find the camera center C Find intrinsic K and rotation R
Pc = 0
![Page 23: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/23.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
M = KR
P = K[R|t]= K[R|�Rc]
= [M|�Mc]
SVD of P!c is the Eigenvector corresponding to
smallest Eigenvalue
Decomposition of the Camera Matrix
Find the camera center C Find intrinsic K and rotation R
Pc = 0
right upper triangle
orthogonal
![Page 24: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/24.jpg)
P =
2
4p1 p2 p3 p4p5 p6 p7 p8p9 p10 p11 p12
3
5
M = KR
P = K[R|t]= K[R|�Rc]
= [M|�Mc]
RQ decomposition
SVD of P!c is the Eigenvector corresponding to
smallest Eigenvalue
Decomposition of the Camera Matrix
Find the camera center C Find intrinsic K and rotation R
Pc = 0
![Page 25: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/25.jpg)
![Page 26: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/26.jpg)
Simple AR program
1. Compute point correspondences (2D and AR tag)
2. Estimate the pose of the camera P
3. Project 3D content to image plane using P
3D locations of planar marker features are known in advance
3D content prepared in advance
(0,0,0)
(10,10,0)
(0,0,0)(10,10,0)
![Page 27: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/27.jpg)
![Page 28: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/28.jpg)
Do you need computer vision to do this?
![Page 29: 11.3 Pose Estimation16385/s17/Slides/11.3_Pose_Estimation.pdf · Pose Estimation known estimate 3D to 2D correspondences Triangulation estimate known 2D to 2D coorespondences Reconstruction](https://reader034.vdocuments.us/reader034/viewer/2022042305/5ed07ad0b0e09762e52be482/html5/thumbnails/29.jpg)
Structure(scene geometry)
Motion (camera geometry) Measurements
Pose Estimation known estimate 3D to 2D correspondences
Triangulation estimate known 2D to 2D coorespondences
Reconstruction estimate estimate 2D to 2D coorespondences