photo from madelyn adams - brown...
TRANSCRIPT
![Page 1: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/1.jpg)
![Page 2: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/2.jpg)
![Page 3: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/3.jpg)
Photo from Madelyn Adams
![Page 4: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/4.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 5: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/5.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 6: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/6.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 7: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/7.jpg)
James, San Francisco, Aug. 2017
![Page 8: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/8.jpg)
CAMERAS, MULTIPLE VIEWS, AND MOTION
![Page 9: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/9.jpg)
What is a camera?
![Page 10: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/10.jpg)
![Page 11: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/11.jpg)
Camera obscura: dark room
• Known during classical period in China and Greece (e.g., Mo-Ti, China, 470BC to 390BC)
Illustration of Camera Obscura Freestanding camera obscura at UNC Chapel Hill
Photo by Seth Ilys
James Hays
![Page 12: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/12.jpg)
Camera obscura / lucida used for tracing
Lens Based Camera Obscura, 1568
drawingchamber.wordpress.com
Camera lucida
![Page 13: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/13.jpg)
Tim’s Vermeer
Vermeer, The Music Lesson, 1665 Tim Jenison (Lightwave 3D, Video Toaster)
![Page 14: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/14.jpg)
Tim’s Vermeer – video still
![Page 15: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/15.jpg)
First Photograph
Oldest surviving photograph
– Took 8 hours on pewter plate
Joseph Niepce, 1826
Photograph of the first photograph
Stored at UT Austin
Niepce later teamed up with Daguerre, who eventually created Daguerrotypes
![Page 16: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/16.jpg)
Point of observation
Figures © Stephen E. Palmer, 2002
Dimensionality Reduction Machine (3D to 2D)
3D world 2D image
![Page 17: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/17.jpg)
Lake Sørvágsvatn in Faroe Islands
100 metres above sea level
![Page 18: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/18.jpg)
Lake Sørvágsvatn in Faroe Islands
100 30 metres above sea level
amusingplanet.com, thanks to Aaron Gokaslan
![Page 19: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/19.jpg)
![Page 20: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/20.jpg)
![Page 21: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/21.jpg)
Holbein’s The Ambassadors - 1533
![Page 22: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/22.jpg)
Holbein’s The Ambassadors – Memento Mori
![Page 23: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/23.jpg)
Parametric (global) transformations
Transformation T is a coordinate-changing machine:p’ = T(p)
What does it mean that T is global?– T is the same for any point pT can be described by just a few numbers (parameters)
For linear transformations, we can represent T as a matrixp’ = Tp
T
p = (x,y) p’ = (x’,y’)
=
y
x
y
xT
'
'
![Page 24: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/24.jpg)
Common transformations
Translation Rotation Scaling
Affine Perspective
Original
Transformed
Slide credit (next few slides):
A. Efros and/or S. Seitz
![Page 25: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/25.jpg)
Scaling• Scaling a coordinate means multiplying each of its
components by a scalar
• Uniform scaling means this scalar is the same for all components:
2
![Page 26: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/26.jpg)
• Non-uniform scaling: different scalars per component:
Scaling
X 2,
Y 0.5
![Page 27: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/27.jpg)
Scaling
• Scaling operation:
• Or, in matrix form:
byy
axx
=
=
'
'
=
y
x
b
a
y
x
0
0
'
'
scaling matrix S
![Page 28: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/28.jpg)
2-D Rotation
(x, y)
(x’, y’)
x’ = x cos() - y sin()
y’ = x sin() + y cos()
x
y
![Page 29: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/29.jpg)
2-D RotationThis is easy to capture in matrix form:
Even though sin() and cos() are nonlinear functions of ,
– x’ is a linear combination of x and y
– y’ is a linear combination of x and y
What is the inverse transformation?
– Rotation by –
– For rotation matrices
( ) ( )
( ) ( )
−=
y
x
y
x
cossin
sincos
'
'
TRR =−1
R
![Page 30: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/30.jpg)
Basic 2D transformations
TranslateRotate
ShearScale
=
y
x
y
x
y
x
1
1
'
'
−=
y
x
y
x
cossin
sincos
'
'
=
y
x
s
s
y
x
y
x
0
0
'
'
=
110
01y
x
t
t
y
x
y
x
=
1
y
x
fed
cba
y
x
Affine
Affine is any combination of
translation, scale, rotation, and shear
![Page 31: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/31.jpg)
Affine Transformations
Affine transformations are combinations of
• Linear transformations, and
• Translations
Properties of affine transformations:
• Lines map to lines
• Parallel lines remain parallel
• Ratios are preserved
• Closed under composition
=
1
y
x
fed
cba
y
x
=
11001
'
'
y
x
fed
cba
y
x
or
![Page 32: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/32.jpg)
2D image transformations (reference table)
Szeliski 2.1
‘Homography’
![Page 33: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/33.jpg)
Projective Transformations
=
wyx
ihgfedcba
wyx
'''Projective transformations are combos of
• Affine transformations, and
• Projective warps
Properties of projective transformations:
• Lines map to lines
• Parallel lines do not necessarily remain parallel
• Ratios are not preserved
• Closed under composition
• Models change of basis
• Projective matrix is defined up to a scale (8 DOF)
![Page 34: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/34.jpg)
Cameras and World Geometry
How tall is this woman?
Which ball is closer?
How high is the camera?
What is the camera
rotation wrt. world?
James Hays
![Page 35: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/35.jpg)
![Page 36: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/36.jpg)
Let’s design a camera
Idea 1: Put a sensor in front of an object
Do we get a reasonable image?
Slide source: Seitz
sensor
![Page 37: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/37.jpg)
Let’s design a camera
Idea 2: Add a barrier to block most rays– Pinhole in barrier
– Only sense light from one direction.• Reduces blurring.
– In most cameras, this aperture can vary in size.
Slide source: Seitz
sensor
![Page 38: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/38.jpg)
Pinhole camera model
Figure from Forsyth
f
f = Focal length
c = Optical center of the camera
c
Real
object
![Page 39: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/39.jpg)
Projection: world coordinates→image coordinates
Camera
Center
(0, 0, 0)
=
Z
Y
X
P.
.
. f Z Y
=
V
Up
.V
U
Z
fXU *−=
Z
fYV *−=
What is the effect if f and Z are equal?
p = distance from
image center
Image
center
(u0, v0)
![Page 40: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/40.jpg)
Projective Geometry
Length (and so area) is lost.
Which is closer?
Who is taller?
![Page 41: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/41.jpg)
Length and area are not preserved
Figure by David Forsyth
B’
C’
A’
![Page 42: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/42.jpg)
Projective Geometry
Perpendicular?
Parallel?
Angles are lost.
![Page 43: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/43.jpg)
Projective Geometry
What is preserved?
• Straight lines are still straight.
![Page 44: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/44.jpg)
Vanishing points and lines
Parallel lines in the world
intersect in the projected image at a “vanishing point”.
Parallel lines on the same plane in the world converge to vanishing points on a “vanishing line”.
E.G., the horizon.
Vanishing Point Vanishing Point
Vanishing Line
![Page 45: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/45.jpg)
Vanishing points and lines
Vanishingpoint
Vanishingpoint
Vertical vanishingpoint
(at infinity)
Slide from Efros, Photo from Criminisi
![Page 46: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/46.jpg)
Pinhole camera model
Forsyth
f
f = Focal length
c = Optical center of the camera
c
Real
object
![Page 47: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/47.jpg)
Projection: world coordinates→image coordinates
Camera
Center
(0, 0, 0)
=
Z
Y
X
P.
.
. f Z Y
=
V
Up
.V
U
Z
fXU *−=
Z
fYV *−=
What is the effect if f and Z are equal?
p = distance from
image center
Image
center
(u0, v0)
![Page 48: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/48.jpg)
Projective geometry
• 2D point in cartesian = (x,y) coordinates
• 2D point in projective = (x,y,w) coordinates
Idea from www.tomdalling.com
Y
X
Projector
![Page 49: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/49.jpg)
Projective geometry
• 2D point in cartesian = (x,y) coordinates
• 2D point in projective = (x,y,w) coordinates
Y
X
Projector W
Idea from www.tomdalling.com
![Page 50: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/50.jpg)
Varying w
w1 w2 < w1
Projected image becomes smaller.
Y
X
Projector
Y
X
Projector
![Page 51: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/51.jpg)
Projective geometry
• 2D point in projective = (x,y,w) coordinates
– w defines the scale of the projected image.
– Each x,y point becomes a ray!
Y
X
Projector W
![Page 52: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/52.jpg)
Projective geometry
• In 3D, point (x,y,z) becomes (x,y,z,w)
• Perspective is w varying with z:
– Objects far away are appear smaller
B’
C’
![Page 53: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/53.jpg)
Homogeneous coordinates
Converting to homogeneous coordinates
2D (image) coordinates 3D (scene) coordinates
Converting from homogeneous coordinates
2D (image) coordinates 3D (scene) coordinates
![Page 54: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/54.jpg)
Homogeneous coordinates
Scale invariance in projection space
=
=
w
y
wx
kw
ky
kwkx
kw
ky
kx
w
y
x
k
Homogeneous Coordinates
Cartesian Coordinates
E.G., we can uniformly scale the projective space, and it will still
produce the same image -> scale ambiguity
![Page 55: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/55.jpg)
Homogeneous coordinates
• Projective
• Point becomes a line
Song Ho Ahn
To homogeneous
From homogeneous
![Page 56: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/56.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 57: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/57.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 58: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/58.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 59: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/59.jpg)
Slide Credit: Savarese
Camera (projection) matrix
XtRKx =x: Image Coordinates: (u,v,1)
K: Intrinsic Matrix (3x3)
R: Rotation (3x3)
t: Translation (3x1)
X: World Coordinates: (X,Y,Z,1)
Ow
iw
kw
jwR,t
X
x
Extrinsic Matrix
![Page 60: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/60.jpg)
X0IKx =
=
10100
000
000
1z
y
x
f
f
v
u
w
K
Slide Credit: Savarese
Projection matrix
Intrinsic Assumptions
• Unit aspect ratio
• Optical center at (0,0)
• No skew
Extrinsic Assumptions• No rotation
• Camera at (0,0,0)
X
x(0,0,0)
![Page 61: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/61.jpg)
Projection: world coordinates→image coordinates
Camera
Center
(0, 0, 0)
=
Z
Y
X
P.
.
. f Z Y
=
V
Up
.V
U
Z
fXU *−=
Z
fYV *−=
p = distance from
image center
Image
center
(u0, v0)
![Page 62: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/62.jpg)
Remove assumption: known optical center
X0IKx =
=
10100
00
00
1
0
0
z
y
x
vf
uf
v
u
w
Intrinsic Assumptions
• Unit aspect ratio
• No skew
Extrinsic Assumptions• No rotation
• Camera at (0,0,0)
James Hays
K
![Page 63: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/63.jpg)
Remove assumption: equal aspect ratio
X0IKx =
=
10100
00
00
1
0
0
z
y
x
vf
uf
v
u
w y
x
Intrinsic Assumptions• No skew
Extrinsic Assumptions• No rotation
• Camera at (0,0,0)
James Hays
![Page 64: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/64.jpg)
Remove assumption: non-skewed pixels
X0IKx =
=
10100
00
0
1
0
0
z
y
x
vf
usf
v
u
w y
x
Intrinsic Assumptions Extrinsic Assumptions• No rotation
• Camera at (0,0,0)
Note: different books use different notation for parameters James Hays
![Page 65: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/65.jpg)
Oriented and Translated Camera
Ow
iw
kw
jw
t
R
X
x
James Hays
![Page 66: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/66.jpg)
Allow camera translation
XtIKx =
=
1100
010
001
100
0
1
0
0
z
y
x
t
t
t
vf
usf
v
u
w
z
y
x
y
x
Intrinsic Assumptions Extrinsic Assumptions• No rotation
James Hays
![Page 67: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/67.jpg)
3D Rotation of Points
Rotation around the coordinate axes, counter-clockwise:
−
=
−
=
−=
100
0cossin
0sincos
)(
cos0sin
010
sin0cos
)(
cossin0
sincos0
001
)(
z
y
x
R
R
R
p
p’
γ
y
z
Slide Credit: Saverese
x
![Page 68: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/68.jpg)
Allow camera rotation
XtRKx =
=
1100
0
1 333231
232221
131211
0
0
z
y
x
trrr
trrr
trrr
vf
usf
v
u
w
z
y
x
y
x
James Hays
![Page 69: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/69.jpg)
Slide Credit: Savarese
Camera (projection) matrix
Ow
iw
kw
jwR,t
X
x
=
1100
0
1 333231
232221
131211
0
0
z
y
x
trrr
trrr
trrr
vf
usf
v
u
w
z
y
x
y
x
XtRKx =x: Image Coordinates: (u,v,1)
K: Intrinsic Matrix (3x3)
R: Rotation (3x3)
t: Translation (3x1)
X: World Coordinates: (X,Y,Z,1)Extrinsic Matrix
![Page 70: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/70.jpg)
Demo – Kyle Simek
“Dissecting the Camera Matrix”
Three-part blog series
• http://ksimek.github.io/2012/08/14/decompose/
• http://ksimek.github.io/2012/08/22/extrinsic/
• http://ksimek.github.io/2013/08/13/intrinsic/
“Perspective toy”
• http://ksimek.github.io/perspective_camera_toy.html
![Page 71: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/71.jpg)
Orthographic Projection
• Special case of perspective projection
– Distance from the COP to the image plane is infinite
– Also called “parallel projection”
– What’s the projection matrix?
Image World
Slide by Steve Seitz
=
11000
0010
0001
1z
y
x
v
u
w
![Page 72: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/72.jpg)
Things to remember
Vanishing points and vanishing lines
Pinhole camera model and camera projection matrix
Homogeneous coordinates
Vanishingpoint
Vanishingline
Vanishingpoint
Vertical vanishingpoint
(at infinity)
XtRKx =
James Hays
![Page 73: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/73.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 74: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/74.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 75: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/75.jpg)
By Suren Manvelyan, http://www.surenmanvelyan.com/gallery/7116
![Page 76: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/76.jpg)
Indy
![Page 77: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/77.jpg)
![Page 78: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/78.jpg)
How to calibrate the camera?(also called “camera resectioning”)
=
1****
****
****
Z
Y
X
w
wv
wu
XtRKx =
James Hays
𝐱 = 𝐌𝐗
Linear least-squares regression!
![Page 79: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/79.jpg)
Simple example: Fitting a line
![Page 80: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/80.jpg)
Least squares line fitting•Data: (x1, y1), …, (xn, yn)
•Line equation: yi = m xi + b
•Find (m, b) to minimize
022 =−= yAApA TT
dp
dE
2
11
1
2
1
1
1
−
=
−
= =
nn
n
i ii
y
y
b
m
x
x
yb
mxE
=−−=
n
i ii bxmyE1
2)(
(xi, yi)
y=mx+b
( ) yAAApyAApA TTTT 1−==
Matlab: p = A \ y;Python: p = np.linalg.lstsq(A,y)[0]
Modified from S. Lazebnik
)()()(2 ApApyApyy TTT +−=
2yAp −=
(Closed form solution)
![Page 81: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/81.jpg)
Example: solving for translation
A1
A2 A3B1
B2 B3
Given matched points in {A} and {B}, estimate the translation of the object
+
=
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
![Page 82: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/82.jpg)
Example: solving for translation
A1
A2 A3B1
B2 B3
Least squares setup
+
=
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
(tx, ty)
−
−
−
−
=
A
n
B
n
A
n
B
n
AB
AB
y
x
yy
xx
yy
xx
t
t
11
11
10
01
10
01
![Page 83: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/83.jpg)
Example: discovering rot/trans/scale
A1
A2 A3
Given matched points in {A} and {B}, estimate the transformation matrix
+
=
y
x
A
i
A
i
B
i
B
i
t
t
y
x
y
x
=
dc
ba
![Page 84: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/84.jpg)
Are these transformations enough?
![Page 85: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/85.jpg)
World vs Camera coordinates
James Hays
![Page 86: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/86.jpg)
Calibrating the Camera
Use an scene with known geometry
– Correspond image points to 3d points
– Get least squares solution (or non-linear solution)
=
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
su
Known 3d
world locationsKnown 2d
image coords
Unknown Camera Parameters
James Hays
M
![Page 87: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/87.jpg)
How do we calibrate a camera?
312.747 309.140 30.086
305.796 311.649 30.356
307.694 312.358 30.418
310.149 307.186 29.298
311.937 310.105 29.216
311.202 307.572 30.682
307.106 306.876 28.660
309.317 312.490 30.230
307.435 310.151 29.318
308.253 306.300 28.881
306.650 309.301 28.905
308.069 306.831 29.189
309.671 308.834 29.029
308.255 309.955 29.267
307.546 308.613 28.963
311.036 309.206 28.913
307.518 308.175 29.069
309.950 311.262 29.990
312.160 310.772 29.080
311.988 312.709 30.514
880 214
43 203
270 197
886 347
745 302
943 128
476 590
419 214
317 335
783 521
235 427
665 429
655 362
427 333
412 415
746 351
434 415
525 234
716 308
602 187
Known 3d
world locations
Known 2d
image coords
James Hays
![Page 88: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/88.jpg)
What is least squares doing?
• Given 3D point evidence, find best M which minimizes error between estimate (p’) and known corresponding 2D points (p).
Camera
center
=
Z
Y
X
P.
.. f Z
Y
=
v
up
.v
u
p = distance from image center
Error between
M estimate
and known
projection point
p’ under M .
![Page 89: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/89.jpg)
What is least squares doing?
• Best M occurs when p’ = p, or when p’ – p = 0
• Form these equations from all point evidence
• Solve for model via closed-form regression
Camera
center
=
Z
Y
X
P.
.. f Z
Y
=
v
up
.v
u
p = distance from image center
Error between
M estimate
and known
projection point
p’ under M .
![Page 90: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/90.jpg)
=
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
su
14131211 mZmYmXmsu +++=
24232221 mZmYmXmsv +++=
34333231 mZmYmXms +++=
Known 3d
locations
Known 2d
image coords
Unknown Camera Parameters
34333231
14131211
mZmYmXm
mZmYmXmu
+++
+++=
34333231
24232221
mZmYmXm
mZmYmXmv
+++
+++=
James Hays
Two equations
per 3D point
correspondence
First, work out
where X,Y,Z
projects to under
candidate M.
![Page 91: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/91.jpg)
=
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
suKnown 3d
locations
Known 2d
image coords
34333231
14131211
mZmYmXm
mZmYmXmu
+++
+++=
34333231
24232221
mZmYmXm
mZmYmXmv
+++
+++=
1413121134333231 )( mZmYmXmumZmYmXm +++=+++
2423222134333231 )( mZmYmXmvmZmYmXm +++=+++
1413121134333231 mZmYmXmumuZmuYmuXm +++=+++
2423222134333231 mZmYmXmvmvZmvYmvXm +++=+++
Unknown Camera Parameters
Next, rearrange into form
where all M coefficients are
individually stated in terms
of X,Y,Z,u,v.
-> Allows us to form lsq
matrix.
![Page 92: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/92.jpg)
=
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
suKnown 3d
locations
Known 2d
image coords
1413121134333231 mZmYmXmumuZmuYmuXm +++=+++
2423222134333231 mZmYmXmvmvZmvYmvXm +++=+++
umuZmuYmuXmmZmYmXm 34333231141312110 −−−−+++=
vmvZmvYmvXmmZmYmXm 34333231242322210 −−−−+++=
Unknown Camera Parameters
Next, rearrange into form
where all M coefficients are
individually stated in terms
of X,Y,Z,u,v.
-> Allows us to form lsq
matrix.
![Page 93: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/93.jpg)
• Finally, solve for m’s entries using linear least squares
• Method 1 –
=
−−−
−−−
−−−
−−−
n
n
nnnnnnnnn
nnnnnnnnn
v
u
v
u
m
m
m
m
m
m
m
m
m
m
m
ZvYvXvZYX
ZuYuXuZYX
ZvYvXvZYX
ZuYuXuZYX
1
1
33
32
31
24
23
22
21
14
13
12
11
111111111
111111111
10000
00001
10000
00001
Ax=b form
MATLAB:
M = A\b;
M = [M;1];
M = reshape(M,[],3)';
Python Numpy:
M = np.linalg.lstsq(A,b)[0];
M = np.append(M,1)M = np.reshape(M, (3,4))
=
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
suKnown 3d
locations
Known 2d
image coords
Hays
Unknown Camera Parameters
A
x
b
Note: Must reshape M afterwards!
![Page 94: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/94.jpg)
=
134333231
24232221
14131211
Z
Y
X
mmmm
mmmm
mmmm
s
sv
suKnown 3d
locations
Known 2d
image coords
• Or, solve for m’s entries using total linear least-squares
• Method 2 –– Find non-trivial solution (not A=0)
=
−−−−
−−−−
−−−−
−−−−
0
0
0
0
10000
00001
10000
00001
34
33
32
31
24
23
22
21
14
13
12
11
1111111111
1111111111
m
m
m
m
m
m
m
m
m
m
m
m
vZvYvXvZYX
uZuYuXuZYX
vZvYvXvZYX
uZuYuXuZYX
nnnnnnnnnn
nnnnnnnnnn
MATLAB:[U, S, V] = svd(A);
M = V(:,end);
M = reshape(M,[],3)’;
Python Numpy:
U, S, Vh = np.linalg.svd(a)
# V = Vh.T
M = Vh[-1,:]
M = np.reshape(M, (3,4))
Ax=0 form
James Hays
Unknown Camera Parameters
A
x
![Page 95: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/95.jpg)
How do we calibrate a camera?
312.747 309.140 30.086
305.796 311.649 30.356
307.694 312.358 30.418
310.149 307.186 29.298
311.937 310.105 29.216
311.202 307.572 30.682
307.106 306.876 28.660
309.317 312.490 30.230
307.435 310.151 29.318
308.253 306.300 28.881
306.650 309.301 28.905
308.069 306.831 29.189
309.671 308.834 29.029
308.255 309.955 29.267
307.546 308.613 28.963
311.036 309.206 28.913
307.518 308.175 29.069
309.950 311.262 29.990
312.160 310.772 29.080
311.988 312.709 30.514
880 214
43 203
270 197
886 347
745 302
943 128
476 590
419 214
317 335
783 521
235 427
665 429
655 362
427 333
412 415
746 351
434 415
525 234
716 308
602 187
Known 3d
world locations
Known 2d
image coords
James Hays
![Page 96: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/96.jpg)
312.747 309.140 30.086
305.796 311.649 30.356
307.694 312.358 30.418
310.149 307.186 29.298
311.937 310.105 29.216
311.202 307.572 30.682
307.106 306.876 28.660
309.317 312.490 30.230
307.435 310.151 29.318
…..
880 214
43 203
270 197
886 347
745 302
943 128
476 590
419 214
317 335
…
Known 3d world locationsKnown 2d image coords
=
−−−−
−−−−
−−−−
−−−−
0
0
0
0
10000
00001
214086.30214140.309214747.3122141086.30140.309747.3120000
880086.30880140.309880747.31288000001086.30140.309747.312
34
33
32
31
24
23
22
21
14
13
12
11
m
m
m
m
m
m
m
m
m
m
m
m
vZvYvXvZYX
uZuYuXuZYX
nnnnnnnnnn
nnnnnnnnnn
1st point
Projection error defined by two equations – one for u and one for v
(𝑢1, 𝑣1) (𝑋1, 𝑌1, 𝑍1)
![Page 97: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/97.jpg)
312.747 309.140 30.086
305.796 311.649 30.356
307.694 312.358 30.418
310.149 307.186 29.298
311.937 310.105 29.216
311.202 307.572 30.682
307.106 306.876 28.660
309.317 312.490 30.230
307.435 310.151 29.318
…..
880 214
43 203
270 197
886 347
745 302
943 128
476 590
419 214
317 335
…
Known 3d world locationsKnown 2d image coords
=
−−−−
−−−−
−−−−
−−−−
−−−−
−−−−
0
0
0
0
10000
00001
203356.3043649.311203796.3052031356.30649.311796.3050000
43356.3043649.31143796.3054300001356.30649.311796.305
214086.30214140.309214747.3122141086.30140.309747.3120000
880086.30880140.309880747.31288000001086.30140.309747.312
34
33
32
31
24
23
22
21
14
13
12
11
m
m
m
m
m
m
m
m
m
m
m
m
vZvYvXvZYX
uZuYuXuZYX
nnnnnnnnnn
nnnnnnnnnnn
2nd point (𝑢2, 𝑣2) (𝑋2, 𝑌2, 𝑍2)
Projection error defined by two equations – one for u and one for v
![Page 98: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/98.jpg)
How many points do I need to fit the model?
XtRKx =
=
1100
0
1 333231
232221
131211
0
0
z
y
x
trrr
trrr
trrr
v
us
v
u
w
z
y
x
5 6Degrees of freedom?
Think 3:
- Rotation around x
- Rotation around y
- Rotation around z
![Page 99: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/99.jpg)
How many points do I need to fit the model?
XtRKx =
=
1100
0
1 333231
232221
131211
0
0
z
y
x
trrr
trrr
trrr
v
us
v
u
w
z
y
x
5 6Degrees of freedom?
M is 3x4, so 12 unknowns, but projective scale ambiguity – 11 deg. freedom.
One equation per unknown -> 5 1/2 point correspondences determines a solution
(e.g., either u or v).
More than 5 1/2 point correspondences -> overdetermined, many solutions to M.
Least squares is finding the solution that best satisfies the overdetermined system.
Why use more than 6? Robustness to error in feature points.
![Page 100: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/100.jpg)
Calibration with linear method
• Advantages
– Easy to formulate and solve
– Provides initialization for non-linear methods
• Disadvantages
– Doesn’t directly give you camera parameters
– Doesn’t model radial distortion
– Can’t impose constraints, such as known focal length
• Non-linear methods are preferred
– Define error as difference between projected points and measured points
– Minimize error using Newton’s method or other non-linear optimization
James Hays
![Page 101: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/101.jpg)
Can we factorize M back to K [R | T]?
• Yes!
• We can directly solve for the individual entries of K [R | T].
James Hays
![Page 102: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/102.jpg)
James Hays
an = nth
column of A
![Page 103: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/103.jpg)
James Hays
![Page 104: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/104.jpg)
James Hays
![Page 105: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/105.jpg)
Can we factorize M back to K [R | T]?
• Yes!
• We can also use RQ factorization (not QR)– R in RQ is not rotation matrix R; crossed names!
• R (right diagonal) is K
• Q (orthogonal basis) is R.
• T, the last column of [R | T], is inv(K) * last column of M.– But you need to do a bit of post-processing to
make sure that the matrices are valid. See http://ksimek.github.io/2012/08/14/decompose/
James Hays
![Page 106: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/106.jpg)
Recovering the camera center
=
1****
****
****
Z
Y
X
s
sv
su
XtRKx =
=
1100
0
1 333231
232221
131211
0
0
z
y
x
trrr
trrr
trrr
v
us
v
u
w
z
y
x
This is not the camera
center C.
It is –RC, as the point
is rotated before tx, ty,
and tz are added
This is t × K
Q
So K-1 m4 is t
So we need
-R-1 K-1 m4 to get C.
Q is K × R.
So we just need -Q-1 m4
James Hays
m4
t
![Page 107: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/107.jpg)
Estimate of camera center
1.5706 -0.1490 0.2598
-1.5282 0.9695 0.3802
-0.6821 1.2856 0.4078
0.4124 -1.0201 -0.0915
1.2095 0.2812 -0.1280
0.8819 -0.8481 0.5255
-0.9442 -1.1583 -0.3759
0.0415 1.3445 0.3240
-0.7975 0.3017 -0.0826
-0.4329 -1.4151 -0.2774
-1.1475 -0.0772 -0.2667
-0.5149 -1.1784 -0.1401
0.1993 -0.2854 -0.2114
-0.4320 0.2143 -0.1053
-0.7481 -0.3840 -0.2408
0.8078 -0.1196 -0.2631
-0.7605 -0.5792 -0.1936
0.3237 0.7970 0.2170
1.3089 0.5786 -0.1887
1.2323 1.4421 0.4506
1.0486 -0.3645
-1.6851 -0.4004
-0.9437 -0.4200
1.0682 0.0699
0.6077 -0.0771
1.2543 -0.6454
-0.2709 0.8635
-0.4571 -0.3645
-0.7902 0.0307
0.7318 0.6382
-1.0580 0.3312
0.3464 0.3377
0.3137 0.1189
-0.4310 0.0242
-0.4799 0.2920
0.6109 0.0830
-0.4081 0.2920
-0.1109 -0.2992
0.5129 -0.0575
0.1406 -0.4527
![Page 108: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/108.jpg)
Oriented and Translated Camera
Ow
iw
kw
jw
t
R
![Page 109: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/109.jpg)
ONE DIFFICULT EXAMPLE…
![Page 110: Photo from Madelyn Adams - Brown Universitycs.brown.edu/courses/cs143/2019_Spring/lectures/2019... · 2019. 4. 13. · Parametric (global) transformations Transformation T is a coordinate-changing](https://reader033.vdocuments.us/reader033/viewer/2022061003/60b2243beb684212b84e67ba/html5/thumbnails/110.jpg)
Erik Johansson – The Architect