bundle adjustment - · bob alice x 1 p bob p alice u 'p 'x black: given variables red:...
TRANSCRIPT
![Page 1: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/1.jpg)
Bundle Adjustment
![Page 2: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/2.jpg)
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
Camera & Point Jacobian
![Page 3: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/3.jpg)
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w
Camera & Point Jacobian
![Page 4: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/4.jpg)
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w 2
2
( )
p pp,X
p p
p p
u ww u
u
w w
v u wv v
w
w
f
2
2
( )
X Xp,X
X X
X X
u ww u
u
w w
v u wv v
ww
f
Camera & Point Jacobian
![Page 5: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/5.jpg)
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w 2
2
( )
p pp,X
p p
p p
u ww u
u
w w
v u wv v
w
w
f
Camera & Point Jacobian
![Page 6: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/6.jpg)
v
Bob Alice
X
bobP aliceP
u
pp
X
1
2
geom
2
2
3
2
3
u
P X P X
P X X
u
P
E
x y
Black: given variables Red: unknowns
( )
p,X
u
v
w
f w 2
2
( )
p pp,X
p p
p p
u ww u
u
w w
v u wv v
w
w
f
2
2
( )
X Xp,X
X X
X X
u ww u
u
w w
v u wv v
ww
f
Camera & Point Jacobian
![Page 7: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/7.jpg)
v
Bob Alice
1X
bobP aliceP
u
pp
X
Black: given variables Red: unknowns
Camera & Point Jacobian ( ) ( )
p X
p ,X p ,XJ J J
p Xj i j i
ij ij ij
j i
f f
J1,bobpJ 2 70 1,bobXJ
1,aliceXJ1,alicepJ2 70
![Page 8: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/8.jpg)
v
Bob Alice
1X
bobP aliceP
u
pp
X
Black: given variables Red: unknowns
Camera & Point Jacobian ( ) ( )
p X
p ,X p ,XJ J J
p Xj i j i
ij ij ij
j i
f f
J1,bobpJ 2 70 1,bobXJ
1,aliceXJ1,alicepJ2 70
2X
2,bobpJ 2 70 2,bobXJ
2,aliceXJ2,alicepJ2 70
2 30
2 30
2 30
2 30
![Page 9: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/9.jpg)
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 3x4
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
![Page 10: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/10.jpg)
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
![Page 11: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/11.jpg)
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
J JTsize of : 24x24
x J J J b f(x)-1T T
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
![Page 12: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/12.jpg)
J =
Cam 1 Pt 1 Pt 4
# of unknowns: 3x7+4x3 # of projections: 9 (not all points are visible from cameras)
J JTsize of : 24x24
x J J J b f(x)-1T T
Main computational bottle neck
Cam 2 Cam 3 Pt 2 Pt 3
Pt 1
Pt 2
Pt 3
Pt 4
Camera 1
Camera 2 Camera 3
![Page 13: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/13.jpg)
![Page 14: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/14.jpg)
![Page 15: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/15.jpg)
Camera trajectory
3D point
# of cameras: 3009 # of 3D points: 1,298,317
# of unknowns: 7x3009+3x1,298,317
J JTsize of : 6,167,690x6,167,690
![Page 16: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/16.jpg)
J =
![Page 17: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/17.jpg)
J = pJXJ
Camera 3D point
p XJ J J
![Page 18: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/18.jpg)
J J =T
p XJ J JT T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
![Page 19: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/19.jpg)
J J =T
p XJ J JT T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
![Page 20: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/20.jpg)
J J =T
p XJ J JT T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
![Page 21: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/21.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
![Page 22: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/22.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T T T
T T T- ( )
p p p X p p
X p X X X X
J J J J J epb X
XJ J J J J ef
![Page 23: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/23.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T T T
T T T- ( )
p p p X p p
X p X X X X
J J J J J epb X
XJ J J J J ef
T T
T T
p p p X p
X p X X X
J J I J J ep
XJ J J J I eor
T
p
X
eA B p
B D X eor
![Page 24: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/24.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
T T T
T T T- ( )
p p p X p p
X p X X X X
J J J J J epb X
XJ J J J J ef
Normal equation:
T T
T T
p p p X p
X p X X X
J J I J J ep
XJ J J J I eor
T
p
X
eA B p
B D X eor
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
![Page 25: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/25.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
![Page 26: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/26.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
![Page 27: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/27.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
-1-1 T
T
--
p X
X
e BD epA BD B 0
XB D e
![Page 28: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/28.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
-1-1 T
T
--
p X
X
e BD epA BD B 0
XB D e
-1-1 T -1
-1 T
- -
-
p X
X
p A BD B e BD e
X D e B p
-1 T-A BD BNote: is Schur complement of D
1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
![Page 29: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/29.jpg)
T T
T
TT T
p p p X
X p X X
J J J J A BJ J
B DJ J J J
J J =T
T T - ( )
pJ J J b X
Xf
Normal equation:
T
p
X
eA B p
B D X e
-1 -1
T
- -
p
X
eA B pI BD I BD
B D X0 I 0 I e
-1-1 T
T
--
p X
X
e BD epA BD B 0
XB D e
-1-1 T -1
-1 T
- -
-
p X
X
p A BD B e BD e
X D e B p
-1 T-A BD BNote: is Schur complement of D
1
d
D
dM
-1
1
-1
-1
d
D
dM
Inversion of block diagonal matrix can be efficiently computed.
Small size matrix
![Page 30: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/30.jpg)
![Page 31: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/31.jpg)
![Page 32: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/32.jpg)
![Page 33: Bundle Adjustment - · Bob Alice X 1 P bob P alice u 'p 'X Black: given variables Red: unknowns Camera & Point Jacobian ªºww( ) ( ) «»ªº¬¼ «»¬¼ww pX p ,X p ,X J J J pX](https://reader033.vdocuments.us/reader033/viewer/2022050605/5fac87ed3d5582547c527edd/html5/thumbnails/33.jpg)