camera models and image formation - eng.utah.educs6320/cv_files/lecture2.pdfperspective projection...
TRANSCRIPT
![Page 1: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/1.jpg)
Camera Models and Image Formation
Srikumar Ramalingam
School of Computing
University of Utah
![Page 2: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/2.jpg)
Reference
Most slides are adapted from the following notes:
• Some lecture notes on geometric computer vision (available online) by Peter Sturm
![Page 3: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/3.jpg)
![Page 4: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/4.jpg)
3D Street Art
Image courtesy: Julian Beaver (VisualFunHouse.com)
![Page 5: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/5.jpg)
3D Street Art
Image courtesy: Julian Beaver (VisualFunHouse.com)
![Page 6: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/6.jpg)
3D Street Art
Image courtesy: Julian Beaver (VisualFunHouse.com)
![Page 7: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/7.jpg)
Pixels are permuted and replicated
SceneOutput Image
• Is this a camera?
![Page 8: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/8.jpg)
ECCV'06 Tutorial on General Imaging Design, Calibration and Applications. Sturm, Swaminathan, Ramalingam
Perspective Imaging
Image Detector
Viewpoint
![Page 9: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/9.jpg)
Many Types of Imaging Systems
ECCV'06 Tutorial on General Imaging Design, Calibration and Applications. Sturm, Swaminathan, Ramalingam
![Page 10: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/10.jpg)
Pinhole model
• A camera maps the 3D world to a 2D image.
• Many such mappings or camera models exist.
• A pinhole model is a good approximation for many existing cameras.
![Page 11: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/11.jpg)
Perspective projection
• A pinhole model can be expressed using an optical center and an image plane. We treat the optical center to be the origin of the camera coordinate frame.
• A 3D point gets projected along the line of sight that connects it with the optical center . Its image point is the intersection of this line with the image plane.
optical axis
Camera center (or)Optical center
pixels
(0,0)Principal point
![Page 12: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/12.jpg)
Perspective projection
• We are given the 3D point in the camera coordinate system (exponent “c” denotes the camera coordinate system).
optical axis
Camera center (or)Optical center
pixels
Why do we have the “1”?
(0,0)Principal point
![Page 13: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/13.jpg)
Why homogenous coordinates?
• Allows us to express common transformations in matrix form:
• Simplifies the concepts such as points at infinity, etc.
![Page 14: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/14.jpg)
Perspective projection
• The image point can be computed using similar triangles:
optical axis (0,0)Principal point
![Page 15: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/15.jpg)
Perspective projection
• In homogeneous coordinates (by adding 1 at the end of a vector), these equations can be written in the form of matrix-vector product: “up to a scale” - scalar multiplication does
not change the equivalence.
optical axis
Camera center (or)Optical center
pixels
(0,0)Principal point
![Page 16: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/16.jpg)
From image plane to pixel coordinates
• - density of pixels along direction.
(0,0)Principal point
• - density of pixels along direction.
Pixels need not be squares (especially the older cameras):
![Page 17: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/17.jpg)
• In homogeneous coordinates we have the following for the image point :
From image plane to pixel coordinates
(0,0)Principal point
![Page 18: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/18.jpg)
3D-to-2D projection
3D to image plane projection Image plane to pixel system
![Page 19: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/19.jpg)
World coordinate frame
• We assume that the 3D point is given in the world coordinate system.
• We model the pose of the camera using a 3x1 translation vector and a 3x3 rotation matrix .
• Let us assume that the superscript “m” denotes 3D points in the world coordinate frame, and the transformation to camera frame is given below:
![Page 20: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/20.jpg)
World coordinate frame
• Rewriting the above equation in homogeneous coordinates to denote the mapping from world to camera coordinate frame:
4x4 matrix
![Page 21: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/21.jpg)
Cross-checking
• We want to ensure that the optical center is the origin of the camera coordinate system.
• In the world coordinate system, the optical center is given by .
![Page 22: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/22.jpg)
Rotation matrices
• Rotations are orthonormal matrices:
– their columns are mutually orthogonal 3-vectors of norm 1.
• For a rotation matrix, the determinant value should be equal to +1. For reflection matrix, the determinant value will be -1.
• The inverse of a rotation matrix is its transpose:
![Page 23: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/23.jpg)
Rotation matrices
• Each rotation can be decomposed into three base rotations:
• The Euler angles and are associated with and axes respectively.
![Page 24: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/24.jpg)
Complete Model
• The following 3x3 matrix is the camera calibration matrix:
![Page 25: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/25.jpg)
Projection Matrix
• The 3 × 4 matrix is called projection matrix. It maps 3D points to 2D image points, all expressed in homogeneous coordinates.
![Page 26: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/26.jpg)
Camera Parameters
• Extrinsic parameters – the rotation matrix and the translation vector .
• Intrinsic parameters – that explains what happens inside a camera - and .
![Page 27: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/27.jpg)
Calibration matrix
• We have 4 parameters defined by 5 coefficients.
There used to be a skew parameter in old cameras, that are not necessary in modern cameras.
![Page 28: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/28.jpg)
Reparameterization
Focal lengths in pixels
Principal point in pixels
![Page 29: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/29.jpg)
Calibration Parameters
• Focal length typically ranges to several hundreds of millimeters.
• The photosensitive area of a camera is typically a rectangle with several millimeters side length and the pixel density is usually of the order of one or several hundreds of pixels per mm
• For cameras with well mounted optics, the principal point is usually very close to the center of the photosensitive area.
![Page 30: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/30.jpg)
What is Camera Calibration?
• The task refers to the problem of computing the calibration matrix.
• We compute the focal length, principal point, and aspect ratio.
![Page 31: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/31.jpg)
Why do you need calibration?
• For a given pixel, the camera calibration allows you to know the light ray along which the camera samples the world.
![Page 32: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/32.jpg)
Calibration Toolbox
• https://www.vision.caltech.edu/bouguetj/calib_doc/htmls/example.html
• http://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/camera_calibration.html
MATLAB
OpenCV
![Page 33: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/33.jpg)
Visual SFM
• http://ccwu.me/vsfm/• https://www.youtube.com/watch?v=SHa_LBIzDac
![Page 34: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/34.jpg)
Forward and backward projection
• Forward: The projection of a 3D point on the image:
![Page 35: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/35.jpg)
• Backward projection: Given a pixel in the image, we determine the set of points in space that map to this point.
Forward and backward projection
Optical Center
![Page 36: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/36.jpg)
Pose estimation
• Given an image of an object whose structure is perfectly known, is it possible find the position and orientation of the camera?
![Page 37: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/37.jpg)
Pose estimation
Camera center and orientation
Camera center and orientation
Compute the pose using constraints from the 3D points
![Page 38: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/38.jpg)
Where do you find applications for pose estimation?
Image courtesy: Oculus Rift Google Tango
Microsoft Hololens
![Page 39: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/39.jpg)
Magic Leap
Where do you find applications for pose estimation?
![Page 40: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/40.jpg)
Pose estimation
What other constraints can you use to find the pose?
Camera Center
![Page 41: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/41.jpg)
Pose estimation
Camera Center
We are given 3 points on the image and their corresponding 3D points.
We compute the 2D distances between pairs of 2D points on the image: and
![Page 42: Camera Models and Image Formation - eng.utah.educs6320/cv_files/Lecture2.pdfPerspective projection • The image point can be computed using similar triangles: optical axis (0,0) Principal](https://reader031.vdocuments.us/reader031/viewer/2022022500/5aa33bc97f8b9ac67a8e1ae3/html5/thumbnails/42.jpg)
Thank You!