![Page 1: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/1.jpg)
Class 3 1
Multi-linear Systems and Invariant Theory
in the Context of Computer Vision and Graphics
Class 3: Infinitesimal Motion
CS329Stanford University
Amnon Shashua
![Page 2: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/2.jpg)
Class 3 2
Material We Will Cover Today
• Infinitesimal Motion Model
• Infinitesimal Planar Homography (8-parameter flow)
• Factorization Principle for Motion/Structure Recovery
• Direct Estimation
![Page 3: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/3.jpg)
Class 3 3
]ˆ)[(sinˆˆ)cos1()(cos wwwIR T w
Infinitesimal Motion Model
0 d
1)cos( d
0)sin( d
][]ˆ)[( wIwdIR
Rodriguez Formula:
![Page 4: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/4.jpg)
Class 3 4
Z
Y
X
PtRPP '
Infinitesimal Motion Model
tPwI x )][(
PPdt
dPP '
tPwP x ][
![Page 5: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/5.jpg)
Class 3 5
X
Y
Z
x
y
Z
Y
X
P
),( 00 yx
),0,0( f
0xZ
Xfx
0yZ
Yfy
Reminder:
Assume: 0,1 00 yxf
PZ
p1
![Page 6: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/6.jpg)
Class 3 6
Infinitesimal Motion Model
tPwP x ][
ZxXZZ
XZZX
Z
X
dt
d
dt
dxu
12
Txs ),0,1( Let PsZ
u T 1
pwstsZ
u xTT ][
1
![Page 7: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/7.jpg)
Class 3 7
Infinitesimal Motion Model
Txs ),0,1(
pwstsZdt
dxu x
TT ][1
Tyr ),1,0(
Tyxp )1,,(
pwrtrZdt
dyv x
TT ][1
![Page 8: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/8.jpg)
Class 3 8
Infinitesimal Planar Motion(the 8-parameter flow)
1PnT 1 cZbYaX
Zcbyax
1 0Z
pwstscbyaxu xTT ][)(
pwrtrcbyaxv xTT ][)(
![Page 9: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/9.jpg)
Class 3 9
Infinitesimal Planar Motion(the 8-parameter flow)
pwstscbyaxu xTT ][)(
pwxxttcbyax x])[,0,1())(( 31
yctbtxwatctwv )()()( 323221
23213 )()( xatwxywbt
ywbtxctatctwu )()()( 313112
xyatwywbt )()( 322
13
![Page 10: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/10.jpg)
Class 3 10
Infinitesimal Planar Motion(the 8-parameter flow)
287321 xxyyxu
278654 yxyyxv
Note: unlike the discrete case, there is no scale factor
![Page 11: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/11.jpg)
Class 3 11
Reconstruction of Structure/Motion(factorization principle)
pwstsZ
u xTT ][
1
pwrtrZ
v xTT ][
1
)()(][ spwpwspws TTx
T
Note:
][][][)( cabbcaabccbaT
][][][ acbcbabac
2 interchanges
1 interchanges
![Page 12: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/12.jpg)
Class 3 12
Reconstruction of Structure/Motion(factorization principle)
pwstsZ
u xTT ][
1
pwrtrZ
v xTT ][
1
)(][ spwpws Tx
T
t
w
rZ
rp
sZ
sp
v
u
T
T
62
1
1
![Page 13: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/13.jpg)
Class 3 13
Reconstruction of Structure/Motion(factorization principle)
Let ),( ijij vu be the “flow” of point i at image j (image 0 is ref frame)
mm
m
n
Tnnn
T
Tnnn
T
nmnn
m
nmnn
m
tt
ww
rZrp
rZrp
sZsp
sZsp
vvv
vvvuuu
uuu
61
1
62
111
111
21
11211
21
11211
...
...
)/1(
..
..
)/1(
)/1(
..
..
)/1(
.
.
...
.
.
.
.
.
....
.
.
...
.
.
.
.
.
....
SMMS
S
V
UW
y
x
![Page 14: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/14.jpg)
Class 3 14
Reconstruction of Structure/Motion(factorization principle)
SMMS
S
V
UW
y
x
Given W, find S,M
Let KLW (using SVD)
SMLAKA ))(( 1 for some 66A
Goal: find 66A such that SKA
using the “structural” constraints on S
![Page 15: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/15.jpg)
Class 3 15
Reconstruction of Structure/Motion(factorization principle)
Goal: find 66A such that SKA
using the “structural” constraints on S
Columns 1-3 of S are known, thus columns 1-3 of A can be determined.
Columns 4-6 of A contain 18 unknowns:
TT rZsZ )/1(,)/1( eliminate Z and one obtains 5 constraints
![Page 16: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/16.jpg)
Class 3 16
Reconstruction of Structure/Motion(factorization principle)
Goal: find 66A such that SKA
using the “structural” constraints on S
62
ny
x
K
KKLet ],...,[ 61 AAA
0,0 45 AKAK yx because
),1,0)(/1(),,0,1)(/1( yzxZ
![Page 17: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/17.jpg)
Class 3 17
Reconstruction of Structure/Motion(factorization principle)
0,0 45 AKAK yx because
),1,0)(/1(),,0,1)(/1( yzxZ
54 AKAK yx
iix
ix xAK
AK
)(
)(
4
6
iiy
iy yAK
AK
)(
)(
5
6
Each point provides 5 constraints,thus we need 4 points and 7 views
![Page 18: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/18.jpg)
Class 3 18
Direct Estimation
),(2 yxgI
),(1 yxfI The grey values of images 1,2
vy
ux
y
x
Goal: find u,v per pixel
Ryx
yxfvyuxgvuS),(
2)],()ˆ,ˆ([)ˆ,ˆ(
![Page 19: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/19.jpg)
Class 3 19
Direct Estimation
vy
ux
y
x
Ryx
yxfvyuxgvuS),(
2)],()ˆ,ˆ([)ˆ,ˆ(
Assume: )ˆ,ˆ(minarg),( ˆ,ˆ vuSvu vu
u
v
)ˆ,ˆ( vuS
v
u
We are assuming that (u,v) can befound by correlation principle (minimizingthe sum of square differences).
![Page 20: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/20.jpg)
Class 3 20
Direct Estimation
Ryx
yxfvyuxgvuS),(
2)],()ˆ,ˆ([)ˆ,ˆ(
Taylor expansion:
)(),(),(),(),( 2 Oyxvgyxugyxgvyuxg yx
Ryx
tyx IgvguvuS),(
2]ˆˆ[)ˆ,ˆ(
),(),(),( yxgyxfyxI t
![Page 21: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/21.jpg)
Class 3 21
Direct Estimation
Ryx
tyx IIvIuvuS),(
2]ˆˆ[)ˆ,ˆ(
tI
yx II ,
image 1 minus image 2
gradient of image 2
Ryx
tyxvu IvIuI),(
2, ][min
yx
yx
II
II
A
.
.
.
.
.
.
v
ux
t
t
I
I
b
.
.
.
![Page 22: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/22.jpg)
Class 3 22
Direct Estimation
Ryx
tyxvu IvIuI),(
2, ][min
2||min bAxx
bAAxA TT
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
“aperture problem” 1)( AArank T
![Page 23: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/23.jpg)
Class 3 23
Direct Estimation
Estimating parametric flow:
287321 xxyyxu
278654 yxyyxv
0(....))( 287321 tyx IIIxxyyx
Every pixel contributes one linear equation for the 8 unknowns
![Page 24: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/24.jpg)
Class 3 24
Direct Estimation
Estimating 3-frame Motion:
pwstsZ
u xTT ][
1
pwrtrZ
v xTT ][
1
Combine with: 0 tyx IvIuI
0][)()(1
tT
yxT
yx IpwrIsItrIsIZ
![Page 25: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/25.jpg)
Class 3 25
Direct Estimation
0][)()(1
tT
yxT
yx IpwrIsItrIsIZ
Let
xy
yxx
yyx
yIxI
xyIIxI
IIyxyI
hpq 2
2
yx
y
x
yx
yIxI
I
I
rIsIh
01
tTT Iqwth
Z
![Page 26: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/26.jpg)
Class 3 26
Direct Estimation
01
tTT Iqwth
Z
0'''1
tTT Iqwth
Z
image 1 to image 2
image 1 to image 3
0]''['' qwttwhthIthI TTTTt
Tt
Each pixel contributes a linear equation to the 15 unknown parameters
![Page 27: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/27.jpg)
Class 3 27
Direct Estimation: Factorization
Let ),( ijij vu
mm
m
n
Tnnn
T
Tnnn
T
nmnn
m
nmnn
m
tt
ww
rZrp
rZrp
sZsp
sZsp
vvv
vvvuuu
uuu
61
1
62
111
111
21
11211
21
11211
...
...
)/1(
..
..
)/1(
)/1(
..
..
)/1(
.
.
...
.
.
.
.
.
....
.
.
...
.
.
.
.
.
....
SMMS
S
V
UW
y
x
be the “flow” of point i at image j (image 0 is ref frame)
![Page 28: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/28.jpg)
Class 3 28
Direct Estimation: Factorization
02
2
t
mnnnyx IV
UII
),...,(1 nxxx IIdiagI
),...,(1 nyyy IIdiagI
mn
mtt
mtt
t
nnII
II
I
....
.
.
.
.
.
.
.
....
1
1
11
![Page 29: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/29.jpg)
Class 3 29
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
Direct Estimation: Factorization
Recall:
ij
ij
ij
ij
ii
ii
h
g
v
u
cb
ba
![Page 30: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/30.jpg)
Class 3 30
mnmnnnH
G
V
U
CB
BA
2222
Direct Estimation: Factorization
),...,( 1 naadiagA
),...,( 1 nbbdiagB
),...,( 1 nccdiagC mnmnn
m
gg
gg
G
....
.
.
.
.
.
.
.
....
1
111
mnmnn
m
hh
hh
H
....
.
.
.
.
.
.
.
....
1
111
![Page 31: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/31.jpg)
Class 3 31
mnmnnnH
G
V
U
CB
BA
2222
Direct Estimation: Factorization
Rank=6 Rank=6
Enforcing rank=6 constraint on the measurement matrix
H
G
removes errors in a least-squares sense.
![Page 32: Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics](https://reader036.vdocuments.us/reader036/viewer/2022062516/56812c07550346895d9076e9/html5/thumbnails/32.jpg)
Class 3 32
mnmnnnH
G
V
U
CB
BA
2222
Direct Estimation: Factorization
H
G
CB
BA
V
U#
Once U,V are recovered, one can solve for S,M as before.