lecture9 camera calibration

31
Camera Calibration

Upload: zukun

Post on 19-May-2015

587 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Lecture9 camera calibration

Camera Calibration

Page 2: Lecture9 camera calibration

What is Camera Calibration?

• Primarily, finding the quantities internal to the camera that affect the imaging process§ Position of image center in the image

• It is typically not at (width/2, height/2) of image

§ Focal length§ Different scaling factors for row pixels

and column pixels§ Skew factor§ Lens distortion (pin-cushion effect)

Page 3: Lecture9 camera calibration

Motivation• Good calibration is important when we need

to § Reconstruct a world model: Virtual L.A. project§ Interact with the world

• Robot, hand-eye coordination

Image plane

We see a square of known size

Evaluation of position of a square for 2 focal lengths (red and blue projection geometry)

Page 4: Lecture9 camera calibration

Scaling of Rows and Columns in Image

• Camera pixels are not necessarily square• Camera output may be analog (NTSC)• Image may be obtained by digitizing card§ A/D converter samples NTSC signal

Camera NTSC signal DigitizingMonitor display

CCD/CMOS

Page 5: Lecture9 camera calibration

Compound Lens Imaging

• Inexpensive single lens systems distort image at its periphery

• Compound lenses may be used to reduce chromatic effects and pin-cushion effects

C

CameraImage plane

Image plane for equivalent pinhole camera is not camera image plane

Principal planes

Center of Projection

Nodal Point

f

f

Page 6: Lecture9 camera calibration

Variety of Techniques

• VERY large literature on the subject• Work of Roger Tsai influential• Linear algebra method described here§ Can be used as initialization for iterative non

linear methods.

• Some interesting methods use vanishing points

Page 7: Lecture9 camera calibration

Camera and Calibration Target

Page 8: Lecture9 camera calibration

Calibration Procedure

• Calibration target: 2 planes at right angle with checkerboard patterns (Tsai grid)§ We know positions of pattern corners only with

respect to a coordinate system of the target§ We position camera in front of target and find

images of corners§ We obtain equations that describe imaging and

contain internal parameters of camera• As a side benefit, we find position and orientation of

camera with respect to target (camera pose)

Page 9: Lecture9 camera calibration

Image Processing of Image of Target

• Canny edge detection• Straight line fitting to detected linked edges• Intersecting the lines to obtain the image

corners• Matching image corners and 3D target

checkerboard corners § By counting if whole target is visible in image

• We get pairs (image point)--(world point)),,(),( iiiii ZYXyx →

Page 10: Lecture9 camera calibration

Central Projection

s

si

s

si

zy

fy

zx

fx

=

=

Scene point(xs , ys , zs )

Image point(xi , yi , f)x

z

Cf

ycenter of projection

Image plane

=

1000

1000000

s

s

s

zyx

ff

wvu

wvywux ii /,/ ==

If world and image points are represented by homogeneous vectors, central projection is a linear transformation:

Page 11: Lecture9 camera calibration

Transformation From Lengths to Pixels

s

si

s

si

zy

fy

zx

fx

=

=

Image point(xi , yi , f)

xpix

ypix

s

ssyiypix

s

ssxixpix

zyzy

kfyyky

zxzx

kfxxkx

00

00

+=+=

+=+=

y

x

=

1000

1000

0

'''

0

0

s

s

s

y

x

zyx

yx

wvu

αα

'/'

'/'

wvy

wux

pix

pix

=

=then

yy

xx

kfkf

==

αα

with

Image center

x0

y0

Transformation uses:

• image center (x0, y0)

• scaling factors kx and ky

Page 12: Lecture9 camera calibration

Internal Camera Parameters

=

1000

1000

'''

0

0

s

s

s

y

x

zyx

yxs

wvu

αα

yy

xx

kfkf

−==

αα

with

• αx and αy “focal lengths” in pixels

• x0 and y0 coordinates of image center in pixels

•Added parameter s is skew parameter

• K is called calibration matrix. Five degrees of freedom.

•K is a 3x3 upper triangular matrix

'/'

'/'

wvy

wux

pix

pix

=

=

[ ]33 0|IK=

=

01000010

0001

1000

00

0

1000 0

0

0

0

y

xs

y

xs

y

x

y

x

αα

αα

Page 13: Lecture9 camera calibration

From Camera Coordinatesto World Coordinates

(Xs , Ys , Zs )

Image point(xi , yi , f)

x

z

C

y

Image plane

Z

X

Y(xs , ys , zs )

OM

Page 14: Lecture9 camera calibration

From Camera Coordinatesto World Coordinates 2

(Xs , Ys , Zs ) in world cordinate system(xs , ys , zs ) in image cordinate system

O

MX

z

x

ki

I

KJ

jC

KJIkjikjiOMCOCM

SSSzyxSSS ZYXTTTzyx +++++=+++=

K.iJ.iI.i SSSxS ZYXTx +++=

+

=

S

S

S

z

y

x

S

S

S

ZYX

TTT

zyx

K.kJ.kI.kK.jJ.jI.jK.iJ.iI.i

Z

CO = T translation vector

Page 15: Lecture9 camera calibration

Homogeneous Coordinates

+

=

S

S

S

z

y

x

S

S

S

ZYX

TTT

zyx

K.kJ.kI.kK.jJ.jI.jK.iJ.iI.i

=

111S

S

S

z

y

x

S

S

S

ZYX

TTT

zyx

000K.kJ.kI.kK.jJ.jI.jK.iJ.iI.i

=

11

1S

S

S

S

S

S

ZYX

zyx

T30

TR

Page 16: Lecture9 camera calibration

From Camera Coordinatesto World Coordinates 3

(Xs , Ys , Zs )(xs , ys , zs )

O

MX

z

x

ki

I

KJ

jC

K)-J)-I)-kjiOC-OMCM

CSCSCSSSS ZZYYXXzyx ((( ++=++=

K.i)-J.i)-I.i)- CSCSCSS ZZYYXXx ((( ++=

)C-(XRxcam~

=

Z

)C-R(T~

=

system coordinate in world expressed OC vector is C~

Page 17: Lecture9 camera calibration

• Here we use instead of T

Homogeneous Coordinates 2

=

11

~

1S

S

S

S

S

S

ZYX

zyx

T30

CR-R

CR- ~

Page 18: Lecture9 camera calibration

Linear Transformation from World Coordinates to Pixels

=

11

~

1S

S

S

S

S

S

ZYX

zyx

T30

CR-R

[ ]

=

1'''

s

s

s

zyx

wvu

33 0|IK

[ ]

=

11

~

'''

S

S

S

ZYX

wvu

T3

33 0CR-R

0|IK

=

1'''

S

S

S

ZYX

wvu

P XPx =

• Combine camera projection and coordinate transformation matrices into a single matrix P

Page 19: Lecture9 camera calibration

• P has 11 degrees of freedom:• 5 from triangular calibration matrix K, 3 from R and 3 from

• P is a fairly general 3 x 4 matrix •left 3x3 submatrix KR is non-singular

Properties of Matrix P

[ ] X0

CR-R0|IKx

T3

33

=

1

~

[ ] [ ] [ ]C-|IRCR-R0

CR-R0|I 3T

333

~~1

~==

[ ]XC-|IRKx 3~

=

[ ]C-|IRKP 3~

=

C~

• Further simplification of P:

Page 20: Lecture9 camera calibration

Calibration

• 1. Estimate matrix P using scene points and their images

• 2. Estimate the interior parameters and the exterior parameters

§ Left 3x3 submatrix of P is product of upper-triangular matrix and orthogonal matrix

[ ]C-|IRKP 3~

=

Page 21: Lecture9 camera calibration

Finding Camera Translation• Find homogeneous coordinates of C in the

scene • C is the null vector of matrix P§ P C = 0:

• Find null vector C of P using SVD§ C is the unit singular vector of P corresponding to the

smallest singular value (the last column of V, where P = U D VT is the SVD of P)

[ ]C-|IRKP 3~

=

=

0000

1100010001

c

c

c

c

c

c

ZYX

ZYX

Page 22: Lecture9 camera calibration

Finding Camera Orientation andInternal Parameters

• Left 3x3 submatrix M of P is of form M=K R§ K is an upper triangular matrix§ R is an orthogonal matrix

• Any non-singular square matrix M can be decomposed into the product of an upper-triangular matrix K and an orthogonal matrix Rusing the RQ factorization§ Similar to QR factorization but order of 2 matrices is

reversed

Page 23: Lecture9 camera calibration

RQ Factorization of M

• Compute

• Multiply M by Rx. The resulting term at (3,2) is zero because of the values selected for c and s

• Multiply the resulting matrix by Ry, after selecting c’ and s’ so that the resulting term at position (3,1) is set to zero

• Multiply the resulting matrix by Rz, after selecting c’’ and s’’ so that the resulting term at position (2,1) is set to zero

−=

−=

−=

1000''''0''''

,'0'

010'0'

,00

001cssc

cs

sc

cssc zyx RRR

2/1233

232

322/12

33232

33

)(,

)( mmm

smm

mc

+=

+−=

RKRRRKMKRRRM Tx

Ty

Tzzyx ==⇒=

Page 24: Lecture9 camera calibration

Computing Matrix P

• Use corresponding image and scene points§ 3D points Xi in world coordinate system§ Images xi of Xi in image

• Write xi = P Xi for all i• Similar problem to finding projectivity

matrix H (i.e. homography) in homework

Page 25: Lecture9 camera calibration

Improved Computation of P

• xi = P Xi involves homogeneous coordinates, thus xi and P Xi just have to be proportional:

• Let p1T, p2

T, p3T be the 3 row vectors of P

=

iT3

iT2

iT1

i

XpXpXp

XP

0=× ii XPx

−−−

iT1ii

T2i

iT3ii

T1i

iT2ii

T3i

ii

XpXpXpXpXpXp

XPx''''''

vuuwwv

0''

''''

=

−−

−⇒

3

2

1

T4

Tii

Tii

Tii

T4

Tii

Tii

Tii

T4

ppp

0XXX0X

XX0

uvuw

vw vector112 a is

ppp

3

2

1

×

Page 26: Lecture9 camera calibration

Improved Computation of P, cont’d

• Third row can be obtained from sum of u’itimes first row - v’i times second row

• So we get 2 independent equations in 11 unknowns (ignoring scale)

• With 6 point correspondences, we get enough equations to compute matrix P

0''

''''

=

−−

3

2

1

T4

Tii

Tii

Tii

T4

Tii

Tii

Tii

T4

ppp

0XXX0X

XX0

uvuw

vw

A p = 0

Page 27: Lecture9 camera calibration

Solving A p = 0

• Linear system A p = 0• When possible, have at least 5 times as

many equations as unknowns (28 points) • Minimize || A p || with the constraint

|| p || = 1§ P is the unit singular vector of A corresponding

to the smallest singular value (the last column of V, where A = U D VT is the SVD of A)

• Called Direct Linear Transformation (DLT)

Page 28: Lecture9 camera calibration

Improving P Solution with Nonlinear Minimization

• Find p using DLT• Use as initialization for nonlinear

minimization of § Use Levenberg-Marquardt iterative minimization

2),( ii

i PXx∑ d

Page 29: Lecture9 camera calibration

Radial Distortion

• We have assumed that lines are imaged as lines

• Not quite true for real lenses§ Significant error for cheap optics and for short

focal lengths

Scene Image Corrected image

Page 30: Lecture9 camera calibration

Radial Distortion Modeling• In pixel cordinates the correction is written

• Minimize using lines known to be straight(x’,y’) is radial projection of (x,y) on straight line

...1)(

)()(

)()()()(

221

20

20

2

00

00

+++=

−+−=

−=−−=−

rrrL

yyxxr

yyrLyyxxrLxx

c

c

κκ

with(x0, y0)

(x, y)(xc, yc)

∑ −+−=i

2221 )'()'(),( ciicii yyxxf κκ

(x, y)

(x’, y’)

Distorted lineAnd ideal line:

Page 31: Lecture9 camera calibration

References

• Multiple View Geometry in Computer Vision, R. Hartley and A. Zisserman, Cambridge University Press, 2000, pp. 138-183

• Three-Dimensional Computer Vision: A Geometric Approach, O. Faugeras, MIT Press, 1996, pp. 33-68

• “A Versatile Camera Calibration Technique for 3D Machine Vision”, R. Y. Tsai, IEEE J. Robotics & Automation, RA-3, No. 4, August 1987, pp. 323-344