sebastian thrun and jana kosecha cs223b computer vision, winter 2007 stanford cs223b computer...

Sebastian Thrun and Jana Kosecha CS223B Computer Vision, Winter 2007

Stanford CS223B Computer Vision, Winter 2007

Lecture 4 Camera Calibration

Professors Sebastian Thrun and Jana Kosecka

CAs: Vaibhav Vaish and David Stavens


Today’s Goals

• Calibration: Problem definition• Solution by nonlinear Least Squares • Solution via Singular Value Decomposition• Homogeneous Coordinates• Distortion• Calibration Software


Camera Calibration

FeatureExtraction

PerspectiveEquations


Perspective Projection, Remember?

fZ Z

Xfx

XO

x


Intrinsic Camera Parameters

• Determine the intrinsic parameters of a camera (with lens)

• What are Intrinsic Parameters?

(can you name 7?)


Intrinsic Parameters

fZ

XO yx oo ,center image

yx ss , size pixel

flength focal

Z

Xfx

21, distortion lens kk


Intrinsic Camera Parameters

• Intrinsic Parameters:– Focal Length f

– Pixel size sx , sy

– Image center ox , oy

– (Nonlinear radial distortion coefficients k1 , k2…)

• Calibration = Determine the intrinsic parameters of a camera


Why Intrinsic Parameters Matter

xx

oZ

X

s

fx

Z

Xfx

yy

oZ

Y

s

fy

Z

Yfy


Questions

• Can we determine the intrinsic parameters by exposing the camera to many known objects?

• If so, – How often do we have to see the object?– How many features on the object do we need?


Example Calibration Pattern

Calibration Pattern: Object with features of known size/geometry


Harris Corner Detector(see Assignment 2)


Why Tilt the Board?


Experiment 1: Parallel Board


30cm10cm 20cm

Projective Perspective of Parallel Board

Z

Xfx


Experiment 2: Tilted Board


30cm10cm 20cm

500cm50cm 100cm

Projective Perspective of Tilted Board


Intrinsics and Extrinsics

• Intrinsics: – Focal Length f

– Pixel size sx , sy

– Image center ox , oy

• Extrinsics:– Location and orientation of k-th calib. pattern:

T,,,


Perspective Camera Model

• Step 1: Transform into camera coordinates

• Step 2: Transform into image coordinates

),,,,(~

~

~

T

Z

Y

X

f

Z

Y

X

W

W

W

C

C

C

yc

c

yim

xc

c

xim

oZ

Y

s

fy

oZ

X

s

fx

~

~

~

~





yc

c

yim

xc

c

xim

oZ

Y

s

fy

oZ

X

s

fx

~

~

~

~

Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

Z

Y

X

cossin0

sincos0

001

cos0sin

010

sin0cos

100

0cossin

0sincos

~

~

~


y

zW

W

W

YW

W

W

y

x

zW

W

W

XW

W

W

x

im

im

o

T

Z

Y

X

T

Z

Y

X

s

f

o

T

Z

Y

X

T

Z

Y

X

s

f

y

x

cossin0

sincos0

001

cos0sin

010

sin0cos

100

cossin0

sincos0

001

cos0sin

010

sin0cos

0cossin

cossin0

sincos0

001

cos0sin

010

sin0cos

100

cossin0

sincos0

001

cos0sin

010

sin0cos

0sincos

The Full Perspective Camera Model


The Calibration Problem

• Given – Calibration pattern with N corners– K views of this calibration pattern

• Recover the intrinsic parameters– We’ll also recover the extrinsics, but we won’t

care about them


Calibration Questions

• Can we determine the intrinsic parameters by exposing the camera to many known objects?

• If so, – How often do we have to see the object?– How many features on the object do we need?– Do we need to see object at angle? Yes.


Today’s Goals



Calibration constraints



yc

c

yim

xc

c

xim

okiZ

kiY

s

fkiy

okiZ

kiX

s

fkix

],[~

],[~

],[

],[~

],[~

],[

][

][

][

][

][

][

][cos][sin0

][sin][cos0

001

][cos0][sin

010

][sin0][cos

100

0][cos][sin

0][sin][cos

],[~

],[~

],[~

kT

kT

kT

iZ

iY

iX

kk

kk

kk

kk

kk

kk

kiZ

kiY

kiX

Z

Y

X

W

W

W

C

C

C

image

feature

k

i


min

][

][

][

][

][cos][sin0

][sin][cos0

001

][cos0][sin

010

][sin0][cos

100

][

][

][

][

][cos][sin0

][sin][cos0

001

][cos0][sin

010

][sin0][cos

0][cos][sin

][

][

][

][

][cos][sin0

][sin][cos0

001

][cos0][sin

010

][sin0][cos

100

][

][

][

][

][cos][sin0

][sin][cos0

001

][cos0][sin

010

][sin0][cos

0][sin][cos

],[

],[

,

2

ki

y

zW

W

W

YW

W

W

y

x

zW

W

W

XW

W

W

x

im

im

o

kT

iZ

iY

iX

kk

kk

kk

kk

kT

iZ

iY

iX

kk

kk

kk

kk

kk

s

f

o

kT

iZ

iY

iX

kk

kk

kk

kk

kT

iZ

iY

iX

kk

kk

kk

kk

kk

s

f

kiy

kix

Camera Calibration


Calibration by nonlinear Least Squares

• Least Mean Square

• Gradient descent:

J

},,,,]},[{]},[{]},[{]},[{{ yxyx oossfkTkkkX

0X

0X

J

)(1.0 11

kkk XX

JXX

min),,,,],[],[],[],[,

][

][

][

(],[

],[

,

2

kiyxyx

W

W

W

im

im oossfkTkkk

iZ

iY

iX

gkiy

kixJ


The Calibration Problem Quiz


• How large would N and K have to be?

• Can we recover all intrinsic parameters?


Intrinsic Parameters, Degeneracy

fZ

XO

yx ss , size pixel

flength focal


Summary Parameters, Revisited

• Extrinsic

– Rotation

– Translation

• Intrinsic

– Focal length

– Pixel size

– Image center coordinates

][],[],[ kkk ][kT

f

),( yx oo

),( yx ssFocal length, in pixel units

Aspect ratioy

x

s

s xs

f




• How large would N and K have to be?

• Can we recover all intrinsic parameters?

N 1 3 1 3 4 4 6

K 1 1 3 3 3 4 6


Constraints

• N points

• K images 2NK constraints

• 4 intrinsics (distortion: +2)

• 6K extrinsics

need 2NK ≥ 6K+4

(N-3)K ≥ 2

Hint: may not be co-linear



N 1 3 1 3 4 4 6

K 1 1 3 3 3 4 6

No No No No Yes Yes Yes

need (N-3)K ≥ 2

Hint: may not be co-linear


Problem with Least Squares

• Many parameters (=slow)

• Many local minima! (=slower)


Today’s Goals






Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

Z

Y

X

cossin0

sincos0

001

cos0sin

010

sin0cos

100

0cossin

0sincos

~

~

~

yc

c

yim

xc

c

xim

oZ

Yfy

oZ

Xfx

~

~

~

~


Calibration Model (extrinsic)

Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

Z

Y

X

cossin0

sincos0

001

cos0sin

01

sin0cos

100

0cossin

0sincos

~

~

~

C

C

CC

C

Y

X

Z

f

Y

X~

~

~

(Homogeneous Coordinates)

(nonlinear perspective projection)


Affine Problem Relaxation

Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

rrr

rrr

rrr

Z

Y

X

333231

232221

131211

~

~

~

Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

Z

Y

X

cossin0

sincos0

001

cos0sin

010

sin0cos

100

0cossin

0sincos

~

~

~


Affine Problem Relaxation

ZWWWC

YWWWC

XWWWC

TZrYrXrZ

TZrYrXrY

TZrYrXrX

333231

232221

131211

~

~

~

Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

rrr

rrr

rrr

Z

Y

X

333231

232221

131211

~

~

~

yc

c

yim

xc

c

xim

oZ

Yfy

oZ

Xfx

~

~

~

~


Calibration via SVD [see Trucco/Verri]

zWWW

xWWW

xx TZrYrXr

TZrYrXrfox

333231

131211

333231

232221

131211

rrr

rrr

rrr

R

zWWW

yWWW

yy TZrYrXr

TZrYrXrfoy

333231

232221

z

y

x

T

T

T

T


Calibration via SVD

N>=7 points, not coplanar

zWi

WW

xWi

WW

xi TZrYrXr

TZrYrXrfx

ii

ii

333231

131211

zWi

WW

yWi

WW

yi TZrYrXr

TZrYrXrfy

ii

ii

333231

232221

)()( 131211232221 xWi

WWxiy

Wi

WWyi TZrYrXrfyTZrYrXrfx iiii


Calibration via SVD

)()( 131211232221 xWi

WWxiy

Wi

WWyi TZrYrXrfyTZrYrXrfx iiii

087654321 vyvZyvYyvXyvxvZxvYxvXx iW

iW

iW

iiW

iW

iW

i iiiiii

y

x

f

f

TxvTv

rvrv

rvrv

rvrv

y

84

137233

126222

115211


Calibration via SVD087654321 vyvZyvYyvXyvxvZxvYxvXx i

Wi

Wi

Wii

Wi

Wi

Wi iiiiii

NWNN

WNN

WNN

WNN

WN

WNN

WWWWWW

WWWWWW

yZyYyXyxZxYxXx

yZyYyXyxZxYxXx

yZyYyXyxZxYxXx

A

NN

2222222222222

1111111111

2

1111

0Av A has rank 7 (without proof)

ion)Decomposit Value(Singular SVD viaTUDVA


Calibration via SVD• Remaining Problem:

• See book

matrixrotation a givet doesn' V

TxvTv

rvrv

rvrv

rvrv

y

84

137233

126222

115211


Summary, SVD Solution

• Replace rotation matrix by arbitrary matrix

• Transform into linear set of equations

• Solve via SVD

• Enforce rotation matrix (see book)

• Solve for remaining parameters (see book)


Comparison

Nonlinear least squares• Gaussian image

noise• Many local minima• Iterative• Can incorporate non-

linear distortion

Singular Value Decomp.• Gaussian parameter

noise (algebraic)• No local minima• “Closed” form• No distortion


Today’s Goals



Homogeneous Coordinates

• Idea: In homogeneous coordinates most operations become linear!

• Extract Image Coordinates by Z-normalization

C

C

CC

C

Y

X

ZY

X~

~

~1

C

C

C

Z

Y

X

~

~

~

1

12

10

000,1

000,12

000,10

5.1

18

15

1

12

10


Today’s Goals



Advanced Calibration:Nonlinear Distortions

• Barrel and Pincushion

• Tangential


Barrel and Pincushion Distortion

telewideangle


Models of Radial Distortion

)1( 42

21 rkrk

y

x

y

x

d

d

distance from center


Tangential Distortion

cheapglue

cheap CMOS chipcheap lens image

cheap camera


Image Rectification (to be continued)


Distorted Camera Calibration

• Set k1k2, solve for undistorted case

• Find optimal k1k2via nonlinear least squares

• Iterate

Tends to generate good calibrations


Today’s Goals

• Calibration: Problem definition

• Solution by nonlinear Least Squares

• Solution via Singular Value Decomposition

• Homogeneous Coordinates

• Distortion

• Calibration Software


Calibration Software: Matlab


Calibration Software: OpenCV


Summary


sebastian thrun and jana kosecha cs223b computer vision, winter 2007 stanford cs223b computer...

Documents