single view metrology - silvio savarese...silvio savarese lecture 4 - 21-feb-14 •review...
TRANSCRIPT
Lecture 4 -Silvio Savarese 21-Feb-14
Professor Silvio Savarese
Computational Vision and Geometry Lab
Lecture 4Single View Metrology
Lecture 4 -Silvio Savarese 21-Feb-14
• Review calibration• Vanishing points and lines• Estimating geometry from a single image• Extensions
Reading:[HZ] Chapter 2 “Projective Geometry and Transformation in 3D”[HZ] Chapter 3 “Projective Geometry and Transformation in 3D”[HZ] Chapter 8 “More Single View Geometry”[Hoeim & Savarese] Chapter 2
Lecture 4Single View Metrology
Calibration Problem
jC
ii PMP
i
i
iv
up
In pixels
TRKM
100
0
cot
sino
o
v
u
K
World ref. system
Calibration Problem
jC
ii PMP
i
i
iv
up
Need at least 6 correspondences
11 unknown
TRKM
In pixelsWorld ref. system
Pinhole perspective projectionOnce the camera is calibrated...
TRKM
C
Ow
-Internal parameters K are known
-R, T are known – but these can only relate C to the calibration rig
Pp
Can I estimate P from the measurement p from a single image?
No - in general [P can be anywhere along the line defined by C and p]
Pinhole perspective projectionRecovering structure from a single view
C
Ow
Pp
unknown known Known/
Partially known/
unknown
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl
Recovering structure from a single view
Transformation in 2D
-Isometries
-Similarities
-Affinity
-Projective
Transformation in 2D
Isometries:
1
y
x
H
1
y
x
10
tR
1
'y
'x
e
- Preserve distance (areas)- 3 DOF- Regulate motion of rigid object
[Euclideans]
Transformation in 2D
Similarities:
1
y
x
H
1
y
x
10
tRs
1
'y
'x
s
- Preserve- ratio of lengths- angles
-4 DOF
Transformation in 2D
Affinities:
1
y
x
H
1
y
x
10
tA
1
'y
'x
a
2221
1211
aa
aaA )(RD)(R)(R
y
x
s0
0sD
Transformation in 2D
Affinities:
1
y
x
H
1
y
x
10
tA
1
'y
'x
a
2221
1211
aa
aaA )(RD)(R)(R
y
x
s0
0sD
-Preserve:- Parallel lines- Ratio of areas- Ratio of lengths on collinear lines- others…
- 6 DOF
Transformation in 2D
Projective:
1
y
x
H
1
y
x
bv
tA
1
'y
'x
p
- 8 DOF- Preserve:
- cross ratio of 4 collinear points- collinearity - and a few others…
The cross ratio
1423
2413
PPPP
PPPP
The cross-ratio of 4 collinear points
Can permute the point ordering
1
i
i
i
iZ
Y
X
P
3421
2431
PPPP
PPPP
P1
P2
P3
P4
Lines in a 2D plane
0cbyax
-c/b
-a/b
c
b
a
l
If x = [ x1, x2]T l 0
c
b
a
1
x
xT
2
1
l
x
y
Lines in a 2D plane
Intersecting lines
llx l
l
Proof
lll
lll
0l)ll(
0l)ll(
lx
x
lx
x is the intersecting point
x
y
x
2D Points at infinity (ideal points)
0x,
x
x
x
x 3
3
2
1
c
b
a
l
c
b
a
l '
'
0
a
b
llLet’s intersect two parallel lines:
• In Euclidian coordinates this point is at infinity
• Agree with the general idea of two lines intersecting at infinity
l
l
0
'
'
2
1
x
x
x
'/'/ baba
x
2D Points at infinity (ideal points)
0x,
x
x
x
x 3
3
2
1
c
b
a
l
c
b
a
l '
'
0
0
lT
a
b
cbax
Note: the line l = [a b c]T pass trough the ideal point
So does the line l’ since a b’ = a’ b
l
l
'/'/ baba
x
Lines infinity l
Set of ideal points lies on a line called the line at infinity
How does it look like?
l
1
0
0
l
T
2
1
0
x
x
0
1
0
0
Indeed:
A line at infinity can thought of the set of “directions” of lines in the plane
'
x
0
'
'
a
b
0
''
''
a
b''
x
Projective transformation of a point at infinity
pHp '
bv
tAH
?pH
z
y
x
p
p
p
bv
tA
'
'
'
0
1
1
is it a point at infinity?
…no!
?pH A
0
'
'
0
1
1
0y
x
p
p
b
tA
An affine
transformation
of a point
at infinity is
still a point at
infinity
Projective transformation of a line (in 2D)
lHl T
bv
tAH
?lH T
b
t
t
1
0
0
bv
tAy
xT
is it a line at infinity?
…no!
?lH T
A
1
0
0
1
0
0
1
0
1
0
0
10 TT
TT
At
AtA
Points and planes in 3D
1
x
x
x
x3
2
1
d
c
b
a
0dczbyax
x
y
z
0xxT
How about lines in 3D? • Lines have 4 degrees of freedom - hard to represent in 3D-space
• Can be defined as intersection of 2 planes
Vanishing points
Points where parallel lines intersect in 3D
In 3D, vanishing points are the equivalent of ideal points in 2D
Hp
0
3
2
1
x
x
x
x
4
3
2
1
p
p
p
p
p
Vanishing points
Points where parallel lines intersect in 3D
C
In 3D, vanishing points are the equivalent of ideal points in 2D
0
3
2
1
x
x
x
x
Hp
= direction of the line in 3D
The horizon line
horizon
lHlT
Phor
l
The horizon line
Planes at infinity & vanishing lines
x
y
z
• Parallel planes intersect the plane at infinity in a common
line – the vanishing line ( horizon)
• A set of vanishing lines defines the plane at infinity
• 2 planes are parallel iff their intersections
is a line that belongs to
1
0
0
0
plane at infinity
Vanishing points and their image
c
b
a
c
b
a
K0IKMXv
0
dv K
d = direction of the line
d
C
v
0
c
b
a
xM
d
Vanishing points - example
C2
v2
C1
v1
R,T
star
1
1
1
1
1K
K
v
vd
2
1
2
1
2K
K
v
vd
21R dd R
v1, v2: measurements
K = known and constant
Can I compute R?No rotation around z
1d2d
In 2D
d
R
R
1
1
1
1
1K
K
v
vd
2
1
2
1
2K
K
v
vd
R
v1
v2
Vanishing lines and their images
C
n
Parallel planes intersect the plane at infinity
in a common line – the vanishing line (horizon)
horiz
TK ln
horizl
Lecture 4 -Silvio Savarese 21-Feb-14
• Review calibration• Vanishing points and line• Estimating geometry from a single image• Extensions
Reading:[HZ] Chapter 2 “Projective Geometry and Transformation in 3D”[HZ] Chapter 3 “Projective Geometry and Transformation in 3D”[HZ] Chapter 8 “More Single View Geometry”[Hoeim & Savarese] Chapter 2
Lecture 4Single View Metrology
Estimating geometry & calibrating the camera from a single image
- Recognize the horizon line
- Measure if the 2 lines meet
at the horizon
- if yes, these 2 lines are // in 3D
•Recognition helps reconstruction!
•Humans have learnt this
Are these two lines parallel or not?
Angle between 2 vanishing points
2
T
21
T
1
2
T
1
vvvv
vvcos
C
d1
v1
v2d2
If 90 0vv 2
T
1
1)(
T
KK
Projective transformation of
1)(
T
KK
It is not function of R, T
654
532
421
symmetric
02 zero-skew31
square pixel02
1.
2.
3. 4.
TRKP 1
PP
Tω
Absolute conic
Angle between 2 scene lines
90
0vv 2
T
1
v1v2
1)(
T
KK
Constraint on K
Single view calibration - example
v2
v3
0vv 2
T
1
0vv 3
T
1
0vv 3
T
2
654
532
421
31
02 Compute :
known up to scale
1)(
T
KK K(Cholesky factorization; HZ pag 582)
v2
Once is calculated, we get K:
Single view reconstruction - example
lh
knownKhoriz
TK ln = Scene plane orientation in
the camera reference system
Select orientation discontinuities
Single view reconstruction - example
Recover the structure within the camera reference system
Notice: the actual scale of the scene is NOT recovered
C
- Recognize the horizon line
- Measure if the 2 lines meet
at the horizon
- if yes, these 2 lines are // in 3D
•Recognition helps reconstruction!
•Humans have learnt this
Are these two lines parallel or not?
Lecture 4 -Silvio Savarese 21-Feb-14
• Review calibration• Vanishing points and lines• Estimating geometry from a single image• Extensions
Reading:[HZ] Chapter 2 “Projective Geometry and Transformation in 3D”[HZ] Chapter 3 “Projective Geometry and Transformation in 3D”[HZ] Chapter 8 “More Single View Geometry”[Hoeim & Savarese] Chapter 2
Lecture 4Single View Metrology
Criminisi & Zisserman, 99
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl
Criminisi & Zisserman, 99
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/merton/merton.wrl
La Trinita' (1426)
Firenze, Santa Maria
Novella; by Masaccio
(1401-1428)
La Trinita' (1426)
Firenze, Santa Maria
Novella; by Masaccio
(1401-1428)
http://www.robots.ox.ac.uk/~vgg/projects/SingleView/models/hut/hutme.wrl
Manually select:• Vanishing points and lines;
• Planar surfaces;
• Occluding boundaries;
• Etc..
Single view reconstruction - drawbacks
Automatic Photo Pop-upHoiem et al, 05
Automatic Photo Pop-upHoiem et al, 05…
Automatic Photo Pop-upHoiem et al, 05…
http://www.cs.uiuc.edu/homes/dhoiem/projects/software.html
Software:
Training
Image Depth
Prediction
Planar Surface
Segmentation
Plane Parameter MRF
Connectivity Co-Planarity
Make3D
youtube
Saxena, Sun, Ng, 05…
A software: Make3D
“Convert your image into 3d model”
Single Image Depth ReconstructionSaxena, Sun, Ng, 05…
http://make3d.stanford.edu/
http://make3d.stanford.edu/images/view3D/185
http://make3d.stanford.edu/images/view3D/931?noforward=true
http://make3d.stanford.edu/images/view3D/108
Coherent object detection and scene
layout estimation from a single image Y. Bao, M. Sun, S. Savarese, CVPR 2010,
BMVC 2010
Y. BaoM. Sun
Next lecture:
Multi-view geometry (epipolar geometry)