CS654: Digital Image Analysis

Lecture 8: Stereo Imaging

Recap of Lecture 7

• Inverse perspective transformation and its issues

•Many to one mapping

• Generalized perspective transformation

• Fundamentals of camera calibration

Outline of Lecture 8

• Fundamentals of stereo imaging

• Calculation of disparity

• Search space for point correspondence

• Correlation based correspondence

Camera calibration

𝑝11 𝑋+𝑝12𝑌+𝑝13𝑍+𝑝14−𝑥𝑋 𝑝41−𝑥𝑌 𝑝42−𝑥𝑍𝑝43−𝑥 𝑝44=0

𝑝21𝑋+𝑝22𝑌 +𝑝23 𝑍+𝑝24− 𝑦 𝑋 𝑝41− 𝑦 𝑌 𝑝42− 𝑦 𝑍𝑝43−𝑦 𝑝44=0

….. (1)

….. (2)6 pairs of points are required

and

and

and

and

and

and

Solving for unknowns

𝐶𝑃=0

[𝑋1 𝑌 1 𝑍1 1 0 0 0 0 −𝑥1 𝑋1 −𝑥1𝑌 1 −𝑥1𝑍 1 −𝑥1𝑋 2 𝑌 2 𝑍2 1 0 0 0 0 −𝑥2 𝑋 2 −𝑥2𝑌 2 −𝑥2𝑍 2 −𝑥2⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮𝑋 6 𝑌 6 𝑍6 1 0 0 0 0 −𝑥6 𝑋 6 −𝑥6𝑌 6 − 𝑥6𝑍 6 − 𝑥60 0 0 0 𝑋1 𝑌 1 𝑍1 1 − 𝑦1𝑋 1 − 𝑦1𝑌1 − 𝑦1𝑍1 − 𝑦10 0 0 0 𝑋 2 𝑌 2 𝑍2 1 − 𝑦2𝑋 2 − 𝑦2𝑌2 − 𝑦2𝑍2 − 𝑦2⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮0 0 0 0 𝑋 6 𝑌 6 𝑍6 1 − 𝑦 6𝑋 6 −𝑦 6𝑌6 − 𝑦6𝑍6 − 𝑦6

] [𝑎11𝑎12𝑎13𝑎14𝑎21𝑎22𝑎23𝑎24𝑎41𝑎42𝑎43𝑎44

]=[0⋮000⋮0]

2𝑛×1212×1

12×1

Perspective transformation

P

PI

𝑍 , 𝑧

𝑌 , 𝑦

𝑋 ,𝑥

World co-ordinate

Image plane

𝑥=𝜆𝑋𝜆−𝑍 𝑦=

𝜆𝑌𝜆−𝑍

𝑋=𝑥𝜆

(𝜆−𝑍 ) 𝑌=𝑦𝜆

(𝜆−𝑍 )Two equations, three unknowns

Stereo geometry

Image courtesy: https://en.wikipedia.org/wiki/Epipolar_geometry

Introducing a second imaging plane

𝑃 :(𝑋 ,𝑌 ,𝑍 )

𝑧

𝑃 𝐼 ′𝑦

𝑥

𝑧 ′

𝑦 ′

𝑥 ′

Focal length of C1

Coordinate system for C1Image point w.r.to C1

Coordinate system for C2Image point w.r.to C2

Focal length of C2

Relationship between coordinate systems

[𝑥 ′𝑦 ′𝑧 ′ ]=[𝑟11 𝑟 12 𝑟13𝑟 14 𝑟 15 𝑟16𝑟 17 𝑟 18 𝑟19 ] [

𝑥𝑦𝑧 ]+[𝑡𝑥𝑡𝑦𝑡 𝑧 ]

Coordinates of Camera #2

Rotation matrix

Translation matrix

Coordinates of Camera #1

Assumptions

•World coordinate w.r.to camera #1:

•World coordinate w.r.to camera #2:

• Two cameras are having identical focal length:

• Coordinate of the point w.r.to x-y-z coordinate system

• Coordinate of the point w.r.to x’-y’-z’ coordinate system

Mathematical relationship between points

• For camera #1

• For camera #2

𝑥0𝑥 𝑖

=𝑦 0𝑦 𝑖

=𝜆− 𝑧0𝜆

𝑥0 ′𝑥 𝑖 ′

=𝑦0 ′𝑦 𝑖′

=𝜆−𝑧 0 ′𝜆

Coordinate transformation is required

Rectified camera configuration

• Assume pure translation, without any rotation

[𝑥 ′𝑦 ′𝑧 ′ ]=[1 0 00 1 00 0 1 ][

𝑥𝑦𝑧 ]+[𝛿𝑥00 ]

[𝑥 ′𝑦 ′𝑧 ′ ]=[1 0 00 1 00 0 1 ][

𝑥𝑦𝑧 ]+[ 0𝛿 𝑦0 ]

[𝑥 ′𝑦 ′𝑧 ′ ]=[1 0 00 1 00 0 1 ][

𝑥𝑦𝑧 ]+[ 00𝛿𝑧 ]

Lateral stereo geometry

Axial stereo geometry

Modified camera configuration after lateral shift along x-axis

LEFT

𝑧

𝑥

𝑦

𝝀𝑂 𝐿

𝐶𝐿

𝑧 ′

𝑥 ′

𝑦 ′

𝝀𝑂𝑅

𝐶𝑅

RIGHT

𝛿𝑥

𝑃 (𝑥0 , 𝑦0 ,𝑧 0)

𝑃 𝐿(𝑥𝐿 , 𝑦𝐿) 𝑃 𝑅(𝑥𝑅 , 𝑦𝑅)

Assumption

• : w.r.to x-y-z coordinate system

• : w.r.to x-y-z coordinate system

• : Origin of the left camera coordinate system

• : Origin of the right camera coordinate system

•World coordinate w.r.to left camera is

• : Lateral shift between to cameras

Mathematical relationship

• For camera #1

• For camera #2

𝑥0𝑥 𝐿

=𝑦0𝑦 𝐿

=𝜆− 𝑧0𝜆

𝑥0𝑥𝑅

=𝑦0𝑦𝑅

=𝜆−𝑧 0𝜆

𝑥0+𝛿𝑥𝑥𝑅+𝛿𝑥

=𝑦 0𝑦𝑅

=𝜆−𝑧 0𝜆

Incorrect

Solve for unknowns

𝑥0𝑥 𝐿

=𝜆−𝑧 0𝜆

…….. (1)

𝑦0𝑦 𝐿

=𝜆−𝑧 0𝜆

…….. (2)

𝑥0+𝛿𝑥𝑥𝑅+𝛿𝑥

=𝜆−𝑧0𝜆

…….. (3)

𝑦 0𝑦𝑅

=𝜆−𝑧 0𝜆

…….. (4)

Coordinate of the 3D world point

𝑧 0=𝜆+𝛿𝑥 .𝜆

𝑥𝐿−(𝑥𝑅+𝛿𝑥)

𝑥0=𝛿𝑥 .𝜆 . 𝑥𝐿

𝑥𝐿−(𝑥𝑅+𝛿𝑥)

𝑦 0=𝛿 𝑥 .𝜆 . 𝑦 𝐿

𝑥𝐿−(𝑥𝑅+𝛿𝑥 )

Depth

Disparity

• Denominator term is significant

• Translating the point to the left camera plane

• Relative displacement: disparity

• Object at infinity

• Depth is inversely related to the disparity

Search space for stereo matching

Left Right

N

N

N

N

𝑦0𝑦 𝐿

=𝜆−𝑧 0𝜆

𝑦 0𝑦𝑅

=𝜆−𝑧 0𝜆

Token Based Stereo

• Detect token• Corners, interest point, edges

• Find correspondences

• Interpolate surface

Correlation Based Stereo Methods

• Depth is computed only at tokens and interpolated/ extrapolated to remaining pixel

• Disparity map is constructed based on a correlation measure

|| 1 tt IIAD

tt II 1CC

tt

tt

II

II

.

.1NC

2

1 tt IISSD

Correlation Based Stereo Methods

• Once disparity is available compute depth using

𝑍=𝜆𝐵𝑑 Separation between the cameras disparity

Error

Index of points

Thank youNext Lecture: Image Interpolation