eccv 2010 tutorial nonrigid structure from otionyaser/tutorialslides-integrated-small.pdf ·...
TRANSCRIPT
-
ECCV 2010 TUTORIAL
NONRIGID STRUCTURE FROM MOTION
YASER SHEIKHThe Robotics InstituteCarnegie Mellon UniversityPittsburgh, USAhttp://cs.cmu.edu/~yaser
SOHAIB KHANComputer Vision LabLUMS School of Science & Engineering Lahore, PAKISTANhttp://web.lums.edu.pk/~sohaib
http://www.cs.cmu.edu/~yaser/ECCV2010Tutorial.html
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE
x
z
3D Structure That Deforms Over Time
4D DYNAMIC STRUCTURE
-
IMAGE MOTION
OBJECT MOTION CAMERA MOTION
-
IMAGE MOTION
OBJECT MOTION CAMERA MOTIONAND
-
RIGID STRUCTURE FROM MOTION
-
NONRIGID STRUCTURE FROM MOTION
-
NONRIGID STRUCTURE FROM MOTION
-
NONRIGID STRUCTURE FROM MOTION
-
NONRIGID STRUCTURE FROM MOTION
ONLY ONE VIEW PER 3D CONFIGURATON: ILL-POSED PROBLEM EQUIVALENT TO FINDING 3D FROM SINGLE IMAGE
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
ASSUMPTIONS� Orthographic Camera� At least 3 images� Rigid Scene� Camera Motion� Corresponding points available
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
NOTATION� P 3D points seen in F frames
� Xj = [Xj, Yj, Zj] is jth 3D point 1��j ��P
� xij = [xij, yij] is the projection of Xj in ith frame 1��i ��F
� Pi is the camera projection matrix if the ith frame 1��i ��F
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
orthographicprojection
matrix2D image
point
3D scenepoint
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
TRICK� Choose scene origin to be center of 3D points� Choose image origins to be center of 2D points� Allows us to drop camera translation
2 rows of a 3D rotation
matrix
imageoffset
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
TRICK� Choose scene origin to be center of 3D points� Choose image origins to be center of 2D points� Allows us to drop camera translation
2 rows of a 3D rotation
matrix
imageoffset
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
PROJECTION OF PP 3D POINTS IN ith IMAGE
PROJECTION OF P 3D POINTS IN F IMAGES
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
PROJECTION OF PP 3D POINTS IN F IMAGES
W R
S
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
Image Observations Matrix, W
×
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
Image Observations Matrix, W
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
HOW TO SOLVE FOR Q� Observation: The correct Q will result in an R whose rows
are pair-wise orthonormal
� The ith image results in the following 3 constraints on Q
� Total 3F constraints on 6 terms of QQT
� Can be solved linearly for G = QQT for F ����
ORTHONORMALITYCONSTRAINTS
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
-
FACTORIZATION METHOD FOR RIGID SFMKontsevich et al. 1987, Tomasi and Kanade 1992
-
NONRIGID STRUCTURE3D Structure That Deforms Over Time
RIGID STRUCTURE
-
NONRIGID STRUCTURE3D Structure That Deforms Over Time
RIGID STRUCTURE NONRIGID STRUCTURE
-
NONRIGID STRUCTURE3D Structure That Deforms Over Time
RIGID STRUCTURE NONRIGID STRUCTURE
-
NONRIGID STRUCTURE FROM MOTIONComparison with Rigid Structure from Motion
RIGID SFM NONRIGID SFM
W R
S
W RS
S(1)
S(2)
S(F)
S(3)...
R1R2
R3� � �
RF
-
NONRIGID STRUCTURE FROM MOTION
RIGID SFM� Inputs:
100 pts x 40 sec x 30 fps x 2 (x,y)= 240,000 observations
� Unknowns:100 points x 3 (X,Y,Z) = 300 unknowns
NONRIGID SFM
Explosion of Unknowns
Example: Given a 40 second video with 100 tracked points
� Inputs:100 pts x 40 sec x 30 fps x 2= 240,000 observations
� Unknowns:100 points x 40 sec x 30 fps x 3 = 360,000 unknowns
-
NONRIGID STRUCTURE FROM MOTIONExplosion of Unknowns
IN GENERAL, NRSFM HAS MORE UNKNOWNS THAN CONSTRAINTS
ILL-POSED PROBLEM: Additional assumptions are necessary to constrain the solution.
HOWEVER…Motion is not random: 3D points are often highly correlated in space and timePoints move because an actuator exerts force on them
Hence their acceleration is limited by the actuating forceTherefore, shape does not deform arbitrarily over time
4D STRUCTURE OFTEN LIES IN A LOW DIMENSIONAL SUBSPACE
-
NONRIGID STRUCTURE FROM MOTIONTwo Major Approaches
Shape Basis3D points at each time instant lie in a low dimensional subspace
Trajectory BasisTrajectory of each point over time lies in a low dimensional subspace
-
EXAMPLES OF APPLICATIONSMatch Moving in Movies
Akhter et al. NIPS 2008
-
EXAMPLES OF APPLICATIONSMotion-Capture
Input Video Two views of the reconstruction
Akhter et al. NIPS 2008
-
EXAMPLES OF APPLICATIONSMotion-Capture Cleanup
ReconstructionUnlabeled DataVideoInput Output
Disney Research, Pittsburgh
-
EXAMPLES OF APPLICATIONSTracking in 2D and 3D
Credit: Iain Matthews
-
EXAMPLES OF APPLICATIONSAnimation
Jain et al. SCA 2010
-
EXAMPLES OF APPLICATIONSBrowsing Image Collections
Credit: Hyun Soo Park
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
time
spac
e
DYNAMIC STRUCTURE
S3F×P =
⎡⎢⎢⎢⎣
X11 X12 · · · X1PX21 X22 · · · X2P...
......
XF1 XF2 · · · XFP
⎤⎥⎥⎥⎦
Saturday, September 4, 2010
-
DYNAMIC STRUCTUREUNDER ORTHOGRAPHIC PROJECTION
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦=
W = RX
Saturday, September 4, 2010
-
LINEAR SHAPE MODEL[T. Cootes et al. 91, Bregler et al. 97]
w1 × + w2 × + … + wK × = α1 α2 αk
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦
Saturday, September 4, 2010
-
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦ =
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
LINEAR SHAPE MODEL
5Saturday, September 4, 2010
-
LINEAR SHAPE MODEL RECONSTRUCTION
5 Basis 15 Basis 25 Basis
6Saturday, September 4, 2010
-
LINEAR SHAPE MODEL UNDER ORTHOGRAPHIC PROJECTION
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
X11 · · · X1PX21 X2P...
...XF1 · · · XFP
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦=
=
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
2F x P 2F x 3F (6F) 3F x P
2F x 3F (6F) 3F x 3k 3k x P
Saturday, September 4, 2010
-
KNOWNS VS UNKNOWNS
2F × P6F + (3F × k) + (k × P )
2F × P ≥ 6F + (3F × k) + (k × P )
KNOWNS:UNKNOWNS:
Saturday, September 4, 2010
-
LINEAR SHAPE MODEL
w1 × + w2 × + … + wK × = α1 α2 αk
Y = R(
K∑i=1
ωibi) +T
RIGID COMPONENT
NONRIGID COMPONENT
IDEA: RIGID COMPONENT GETS FOLDED INTO PROJECTION
Saturday, September 4, 2010
-
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Saturday, September 4, 2010
-
BREGLER et al. 2000Nested SVD
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦=
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkRF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦=
2F x 3k 3k x P
Saturday, September 4, 2010
-
BREGLER et al. 2000Outer SVD
⎡⎢⎢⎢⎣
x11 · · · x1Px21 x2P...
...xF1 · · · xFP
⎤⎥⎥⎥⎦=
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkRF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
2F x 3k 3k x P
SVD
W H B=
W = UDVT
W = (UD12 )(D
12VT )
W = ĤB̂
Saturday, September 4, 2010
-
BREGLER et al. 2000Inner SVD
SVD
W = ĤB̂⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkR1
⎤⎥⎥⎥⎦H
[ω11r
11 ω11r
21 ω11r
31 · · · ω1kr11 ω1kr21 ω1kr31
ω11r41 ω11r
51 ω11r
61 · · · ω1kr41 ω1kr51 ω1kr61
]
⎡⎢⎢⎢⎣
ω11r11 ω11r
21 ω11r
31 ω11r
41 ω11r
51 ω11r
61
ω12r11 ω12r
21 ω12r
31 ω12r
41 ω12r
51 ω12r
61
......
ω1kr11 ω1kr
21 ω1kr
31 ω1kr
41 ω1kr
51 ω1kr
61
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11ω12...
ω1k
⎤⎥⎥⎥⎦[r11 r
21 r
31 r
41 r
51 r
61
]
rank 1
=
=h1
h′1 =
h′1 = udvT = ω̂r̂
=
METRIC RECTIFICATION USING ORTHONORMALITY CONSTRAINTS
Saturday, September 4, 2010
-
BREGLER et al. 2000OVERVIEW
• OUTER SVD: PERFORM SVD ON W TO GET ESTIMATES OF:• H: CAMERA PROJECTIONS AND COEFFICIENTS
• INNER SVD: PERFORM SVD ON H TO GET ESTIMATES OF:• OMEGA: COEFFICIENTS• R: CAMERA PROJECTIONS
• METRIC RECTIFY USING ORTHONORMALITY CONSTRAINTS• B: THE SHAPE BASIS
Saturday, September 4, 2010
-
RESULTS
Saturday, September 4, 2010
-
BREGLER et al. 2000IN PERSPECTIVE
• SEMINAL WORK: SHOWED THAT FACTORIZATION METHODS CAN BE APPLIED TO NONRIGID OBJECTS
• CASCADING ERROR: ANY OUTER SVD ESTIMATION ERROR CASCADES INTO INNER SVD ESTIMATION
• AMBIGUITY ERROR: ESTIMATION OF METRIC RECTIFICATION• NUMBER OF BASIS: LARGE NUMBER OF BASIS REQUIRED• MISSING DATA: NEEDS COMPLETE W MATRIX
Saturday, September 4, 2010
-
METRIC RECTIFICATIONAMBIGUITY
W = ĤB̂
W = ĤGG−1B̂
H = ĤG B = G−1B̂
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkR1
⎤⎥⎥⎥⎦H ==
⎡⎣ Ĥ
⎤⎦⎡⎣ | | |g1 g2 . . . gk
| | |
⎤⎦
G3k X 3k
Saturday, September 4, 2010
-
METRIC RECTIFICATIONORTHONORMALITY CONSTRAINT
⎡⎢⎢⎢⎣
ω11R1 · · · ω1kR1ω21R2 ω2kR2
......
ωF1RF · · · ωFkR1
⎤⎥⎥⎥⎦H ==
⎡⎣ Ĥ
⎤⎦⎡⎣ | | |g1 g2 . . . gk
| | |
⎤⎦
⎡⎣ Ĥ
⎤⎦⎡⎣ |gk
|
⎤⎦ =
⎡⎢⎢⎢⎣
ω1kR1ω2kR2
...ωFkRF
⎤⎥⎥⎥⎦
ωikRi = Ĥ2i−1:2igk
RiRTi = I
ORTHONORMALITY CONSTRAINT
H2i−1:2igkgTk Ĥ2i−1:2i = ω2ikI
Saturday, September 4, 2010
-
METRIC RECTIFICATIONORTHONORMALITY CONSTRAINT
RiRTi = I
ORTHONORMALITY CONSTRAINT
H2igkgTk Ĥ2i−1 = 0
H2igkgTk Ĥ2i = ω
2ikH2i−1gkg
Tk Ĥ2i−1 = ω
2ik
H2i−1gkgTk Ĥ2i−1 = H2igkgTk Ĥ2i
H2i−1:2igkgTk Ĥ2i−1:2i = ω2ikI =
[ω2ik 00 ω2ik
]
Saturday, September 4, 2010
-
CHALLENGE?AMBIGUITY
Saturday, September 4, 2010
-
OPTIMIZATION
Saturday, September 4, 2010
-
CHALLENGEMISSING DATA
• A.M. Buchanan and A.W. Fitzgibbon, “Damped Newton Algorithms for Matrix Factorization with Missing Data,” IEEE International Conference on Computer Vision and Pattern Recognition, 2005.
• L. Torresani, A. Hertzmann, and Christoph Bregler, “Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors,” Transactions on Pattern Analysis and Machine Intelligence, 2008.
• SPANISH FOLKS
• CVPR 2010 BEST PAPER
• BRANCH AND BOUND
Saturday, September 4, 2010
-
CHALLENGESOVERVIEW
• MISSING DATA• BEST K• TRILINEAR OPTIMIZATION
Saturday, September 4, 2010
-
LINEAR SHAPE MODEL PERSPECTIVE PROJECTION
Saturday, September 4, 2010
-
LINEAR SHAPE MODEL MAXIMUM LIKELIHOOD SOLUTION
Saturday, September 4, 2010
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE FROM MOTIONTwo Major Approaches
Shape Basis3D points at each time instant lie in a low dimensional subspace
Trajectory BasisTrajectory of each point over time lies in a low dimensional subspace
-
DYNAMIC STRUCTURE
Shape
Traj
ecto
ry
-
DYNAMIC STRUCTURE
Shape
Shape Representation
LINEAR SHAPE MODEL
-
DYNAMIC STRUCTURE
Traj
ecto
ry
Trajectory Representation
LINEAR TRAJECTORY MODEL
X-co
ordi
nate
of
traj
ecto
ry o
f han
d
…+ + +a1 a2 ak=
time
-
DYNAMIC STRUCTURE
Traj
ecto
ry
Trajectory Representation
LINEAR TRAJECTORY MODEL
X-co
ordi
nate
of
traj
ecto
ry o
f han
d
time
kth trajectory basis vector
Trajectory of jth point (X-component only)
Trajectory Coefficient Contribution of kth basis in the trajectory of jth point
-
TRAJECTORY REPRESENTATION OFDYNAMIC STRUCTURE
X-co
ordi
nate
of
traj
ecto
ry
…+ + +a1 a2 ak=
time
-
TRAJECTORY REPRESENTATION OFDYNAMIC STRUCTURE X-component of trajectory
of jth point as linear combination of K basis trajectories
X-component of trajectory of all point as linear combination of K basis trajectories
-
X-component of trajectory of all points
X, Y and Z-components of trajectory of all points
-
TRAJECTORY REPRESENTATION
SStructure
£Basis
ACoefficents
of Dynamic Structure
-
TRAJECTORY REPRESENTATIONof Dynamic Structure Under Orthographic Projection
W RS
R1R2
R3� � �
RF
-
TRAJECTORY REPRESENTATIONof Dynamic Structure Under Orthographic Projection
W R
R1R2
R3� � �
RF£
A
Structure S, in trajectory subspace represented by K trajectory basis
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
W £A= R
W B= R
SHAPE FACTORIZATION
TRAJECTORY FACTORIZATION
Weights
Traj basis
Shape basis
Weights
DUALITYWeights and Bases
Shape weights are trajectory basis and trajectory weights are shape basis
-
����
�
�
����
�
�
�
FPFPFPFFF
PPP
PPP
PF
ZYXZYX
ZYXZYXZYXZYX
�
�
�
�
111
222212121
111111111
*3S
� rank of columns = rank of rows� Shape model and trajectory model has equal compaction
power
������PPP ZYX� 11YY1XX ��
��������
PPP ZYX� 22YY2�
�FPFPFP� �FPFPFP ZYFX F�����
�
YX Z
� ZYX 1111YY11XXZYX 2121YY21
� FFF 111 FFF ZYFX F 111
DUALITYWeights and Bases
-
����
�
�
����
�
�
�
FPFPFPFFF
PPP
PPP
PF
ZYXZYX
ZYXZYXZYXZYX
�
�
�
�
111
222212121
111111111
*3S
PROOF OF DUALITYWeights and Bases
Consider rearranged structure matrix SS*
-
PROOF OF DUALITYWeights and Bases
Consider rearranged structure matrix SS*
where
1st shape basis
-
PROOF OF DUALITYWeights and Bases
To link shape to jjth trajectory, we select the coefficients related to jth point
-
PROOF OF DUALITYWeights and Bases
Can be rewritten as Compare to Trajectory Representation
-
ILLUSTRATION OF DUALITYSVD Shape and Trajectory Basis for Mocap Structure
Shape Coefficients ��������������� Trajectory Coefficients ������������
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
W R£
A
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
-
W R£
A
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
1. Deformation constrained by physical actuation2. Trajectories vary smoothly and not randomly3. Can be compactly represented by predefined basis
e.g. Discrete Cosine Transform
Object Independent Basision
-
DCT BASIS
-
PREDEFINING TRAJECTORY BASIS� We showed that PCA approaches DCT (Discrete
Cosine Transform) on CMU’s body MOCAP database.
67
21st – 26thbasis
1st – 6thbasis
41st – 46thbasis
PCA DCT
-
COMPACTNESS OF DCT BASIS
68
-
DCT RECONSTRUCTION
35 Basis 50 Basis 65 Basis
69
SA \� AS �ˆ
-
W R£
A
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
�
-
WA
ESTIMATING STRUCTURE VIA TRAJECTORY MODEL
�
1. By SVD, compute2. Correct solution differs by a linear transform
3. Solving for QQ ?
Solution
-
FINDING QThe correct Q will yield the correct form of �
We can just estimate first 3 columns of Q instead of estimating full Q
If Q|||is known:� Compute R� Compute �� Compute A
-
FINDING QIIIThe correct Q will yield the correct form of �
Orthonormality Constraints
Each image yields 3 constraints because � is known
F images yield 3F constraintsAt least 3K images needed to constrain the solution
-
RESULTS
-
Traj Basis
Xiao0
2
4
6
8
10
12
QUANTITATIVE RESULTSStructure Error Rotation Error
Traj BasisTorresani
Xiao0
0.2
0.4
0.6
0.8
1
1.2
1.4
Traj Basis
Torresani
Xiao
We use synthetic and Motion captured data for quantitative experiments
-
MOTION CAPTURE DATASETS
Input Video Two views of the reconstruction
DANCE DATASET75 points, 264 frames, K=5
-
Torresani et al. 2005 Xiao et al. 2004
-
MOTION CAPTURE DATASETS
Input Data Two views of the reconstruction
STRETCH DATASET41 points, 370 frames, K=12
-
MOTION CAPTURE DATASETS
Input Data Two views of the reconstruction
PICKUP DATASET41 points, 357 frames, K=12
-
Two views of the reconstruction
CUBES SEQUENCES14 points, 200 frames, K=2
RESULTS ON REAL VIDEOS
-
Two views of the reconstruction
MATRIX SEQUENCE30 points, 93 frames, K=3
RESULTS ON REAL VIDEOS
-
Two views of the reconstruction
PIE DATASET68 points, 240 frames, K=2
RESULTS ON REAL VIDEOS
-
Two views of the reconstruction
DINOSAUR SEQUENCE49 points, 231 frames, K=12
RESULTS ON REAL VIDEOS
-
AS CAMERA MOTION INCREASESAS OBJECT MOTION DECREASES
RECONSTRUCTION STABILITY INCREASES
-
SHAPE MODEL VS. TRAJECTORY MODELShape Trajectory
Model Can be learnt Hard to specialize
Specificity Object dependent Generalize
Ordering of frames Irrelevant Exploited
Ordering of points Exploited Irrelevant
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality
Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand limitations
Shape Estimation
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand Limitations
Shape Estimation
Tuesday, September 7, 2010
-
3D TRAJECTORY ESTIMATIONECCV 2010
Tuesday, September 7, 2010
-
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Tuesday, September 7, 2010
-
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
Tuesday, September 7, 2010
-
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
]
Tuesday, September 7, 2010
-
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
][
xi1
]×Pi
[Xi1
]= 0
Tuesday, September 7, 2010
-
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
][
xi1
]×Pi
[Xi1
]= 0
QiXi = −qi
Tuesday, September 7, 2010
-
xi ∈ R2
Xi ∈ R3
SINGLE VIEW RECONSTRUCTION
4
Single View Reconstruction
si
[xi1
]= Pi
[Xi1
][
xi1
]×Pi
[Xi1
]= 0
QiXi = −qi2x3 3x1 2x1
Tuesday, September 7, 2010
-
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
STRUCTURE FROM MOTION
5
R2
R2
R2
R3
Multiple ViewReconstruction
Tuesday, September 7, 2010
-
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
STRUCTURE FROM MOTION
5
R2
R2
R2
R3
Multiple ViewReconstruction
Tuesday, September 7, 2010
-
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
STRUCTURE FROM MOTION
5
R2
R2
R2
R3
Multiple ViewReconstruction
IDEA: ESTIMATE CAMERA FROM RIGID PART
Tuesday, September 7, 2010
-
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Tuesday, September 7, 2010
-
CHALLENGETRILINEAR ESTIMATION
W = RΩB
⎡⎢⎢⎢⎣
R1R2
. . .
RF
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
ω11 · · · ω1kω21 ω2k...
...ωF1 · · · ωFk
⎤⎥⎥⎥⎦
⎡⎢⎢⎢⎣
−b1−−b2−
...−bk−
⎤⎥⎥⎥⎦
Tuesday, September 7, 2010
-
RECONSTRUCTION EVENTS
7Trajectory Reconstruction
Tuesday, September 7, 2010
-
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
Tuesday, September 7, 2010
-
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
Tuesday, September 7, 2010
-
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
si
[xi1
]= Pi
[X1
]
2Fx3 3x1 2Fx1
Structure from Motion
Tuesday, September 7, 2010
-
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
2Fx3F 3Fx1 2Fx1
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
si
[xi1
]= Pi
[X1
]
2Fx3 3x1 2Fx1
Structure from Motion
Tuesday, September 7, 2010
-
si
[xi1
]= Pi
[Xi1
]
RECONSTRUCTION EVENTS
7
R2
R3
R2
R3
R2
R3
Trajectory Reconstruction
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
2Fx3F 3Fx1 2Fx1
⎡⎢⎣
Q1...
QF
⎤⎥⎦X =
⎡⎢⎣
q1...qF
⎤⎥⎦
si
[xi1
]= Pi
[X1
]
2Fx3 3x1 2Fx1
Structure from Motion
Tuesday, September 7, 2010
-
QX = q
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
Tuesday, September 7, 2010
-
QX = q
=
⎡⎢⎣
Q1. . .
QF
⎤⎥⎦⎡⎢⎣
X1...
XF
⎤⎥⎦ =
⎡⎢⎣
q1...qF
⎤⎥⎦
Tuesday, September 7, 2010
-
⎡⎢⎢⎢⎣
X11X12...
X1F
⎤⎥⎥⎥⎦3F×1
=
⎡⎣ | | |θ1 θ2 · · · θk
| | |
⎤⎦3F×k
⎡⎢⎢⎢⎣
β1β2...βk
⎤⎥⎥⎥⎦k×1
= Θβ
Trajectory Reconstruction
35 Basis 50 Basis 65 Basis
9Tuesday, September 7, 2010
-
⎡⎢⎢⎢⎣
X11X12...
X1F
⎤⎥⎥⎥⎦3F×1
=
⎡⎣ | | |θ1 θ2 · · · θk
| | |
⎤⎦3F×k
⎡⎢⎢⎢⎣
β1β2...βk
⎤⎥⎥⎥⎦k×1
= Θβ
Trajectory Reconstruction
35 Basis 50 Basis 65 Basis
9Tuesday, September 7, 2010
-
⎡⎢⎢⎢⎣
X11X12...
X1F
⎤⎥⎥⎥⎦3F×1
=
⎡⎣ | | |θ1 θ2 · · · θk
| | |
⎤⎦3F×3k
⎡⎢⎢⎢⎣
β1β2...βk
⎤⎥⎥⎥⎦3k×1
= Θβ
X = Θβ
=
Tuesday, September 7, 2010
-
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
Tuesday, September 7, 2010
-
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
R2
R2
R2
R3R
3 R3
Trajectory Reconstruction
Tuesday, September 7, 2010
-
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
R2
R2
R2
R3
Multiple ViewReconstruction
R2
R2
R2
R3R
3 R3
Trajectory Reconstruction
Tuesday, September 7, 2010
-
Trajectory Reconstruction
11
R2
R3
Single View Reconstruction
R2
R2
R2
R3
Multiple ViewReconstruction
R2
R2
R2
R3R
3 R3
Trajectory Reconstruction
ΘLinear
Transform
R2
R2
R2
R3k
Trajectory Reconstruction
Tuesday, September 7, 2010
-
LINEAR SOLUTION
=
QX = q
2Fx3F3Fx1 2Fx1
QΘβ = q
=
2Fx13Fx3k 3kx12Fx3F
Tuesday, September 7, 2010
-
LINEAR SOLUTION
=
QX = q
2Fx3F3Fx1 2Fx1
βkβ2β1 …+ + +=
X = Θβ
=
3Fx3k3Fx1 3kx1
QΘβ = q
=
2Fx13Fx3k 3kx12Fx3F
Tuesday, September 7, 2010
-
ALGORITHM
Tuesday, September 7, 2010
-
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA
Tuesday, September 7, 2010
-
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC
Tuesday, September 7, 2010
-
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
Tuesday, September 7, 2010
-
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
• CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS
Tuesday, September 7, 2010
-
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
• CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS• SOLVE LINEAR SYSTEM FOR DCT COEFFICIENTS
Tuesday, September 7, 2010
-
ALGORITHM
• GIVEN POINT CORRESPONDENCES AND EXIF DATA• ESTIMATE THE CAMERA MATRICES USING RANSAC• USING CAMERA MATRICES AND DYNAMIC POINT
CORRESPONDENCES:
• CREATE OVERLOADED LINEAR SYSTEM USING DCT BASIS• SOLVE LINEAR SYSTEM FOR DCT COEFFICIENTS
• BUNDLE ADJUSTMENT
Tuesday, September 7, 2010
-
MISSING DATA
14
QΘβ = q
=2Fx1
3Fx3k
3kx12Fx3F
Tuesday, September 7, 2010
-
MISSING DATA
14
QΘβ = q
=2Fx1
3Fx3k
3kx12Fx3F
Tuesday, September 7, 2010
-
PARK ET AL., ECCV 20103D Reconstruction of a Moving Point from a Series of 2D Projections
Tuesday, September 7, 2010
-
16Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
NONRIGID STRUCTURE FROM MOTION
Introduction to Nonrigid SFM
Shape Representation
Ambiguity of Orthogonality Constraints
Trajectory Representation
Shape-Trajectory Duality
TrajectoryEstimation
Tutorial Outline
Reconstructibilityand Limitations
Shape Estimation
Tuesday, September 7, 2010
-
AMBIGUITY
20Tuesday, September 7, 2010
-
AMBIGUITY
20
THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) = ±1
Tuesday, September 7, 2010
-
AMBIGUITY
20
THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) =
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
THEOREM 2: limη→inf
β = β̂
±1
Tuesday, September 7, 2010
-
THEOREM 1: Trajectory reconstruction using any linear trajectory basis is impossible if corr(X,C) = ±1
X = aC+ b
C
CORRELATED X and C
Tuesday, September 7, 2010
-
HYPERPLANE OF SOLUTIONS
Single View Reconstruction
Trajectory Reconstruction
X̂ = aX+ (1− a)C X̂ = AX+ (1−A)C
C
X X
C
X̂ X̂
SINGLE VIEW 3D RECONSTRUCTION
MULTIPLE VIEWDYNAMIC 3D
RECONSTRUCTION
Tuesday, September 7, 2010
-
HYPERPLANE OF SOLUTIONS
Single View Reconstruction
Trajectory Reconstruction
X̂ = aX+ (1− a)C X̂ = AX+ (1−A)C
C
X X
C
X̂ X̂
SINGLE VIEW 3D RECONSTRUCTION
MULTIPLE VIEWDYNAMIC 3D
RECONSTRUCTION
Tuesday, September 7, 2010
-
QΘβ = qR3F
GEOMETRY OF C AND X
Trajectory Reconstruction
Tuesday, September 7, 2010
-
QΘβ = q
C
R3F
GEOMETRY OF C AND X
Trajectory Reconstruction
C
Tuesday, September 7, 2010
-
QΘβ = q
X
C
R3F
GEOMETRY OF C AND X
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
-
Θ
Θ⊥
QΘβ = q
X
C
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
-
Θ
Θ⊥
QΘβ = q
X
C
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
-
Θ
Θ⊥
QΘβ = q X̂ = AX+ (1−A)C
X
C
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
Tuesday, September 7, 2010
-
Θ
Θ⊥
QΘβ = q X̂ = AX+ (1−A)C
X
C
X̂
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
-
Θ
Θ⊥
QΘβ = q X̂ = AX+ (1−A)C
X
C
X̂
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
Trajectory Reconstruction
X
C
e
X̂
Tuesday, September 7, 2010
-
X = aC
23
X
C
X̂
Θ
Θ⊥
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
-
X = aC
23
X
C
X̂
Θ
Θ⊥
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
-
Θ
Θ⊥
X = aC
23
X
C
X̂
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
-
Θ
Θ⊥
X = aC
23
X
C
X̂
R3F
THEOREM 1: CORRELATED C AND X
Trajectory Reconstruction
X
C
X̂
Tuesday, September 7, 2010
-
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
26
THEOREM 2: limη→inf
β = β̂
RECONSTRUCTIBILITY
Tuesday, September 7, 2010
-
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
26
THEOREM 2: limη→inf
β = β̂
= ‖Θ⊥β⊥C‖η ∝ HOW POORLY THE BASIS DESCRIBES C
RECONSTRUCTIBILITY
Tuesday, September 7, 2010
-
η =‖Θ⊥β⊥C‖‖Θ⊥β⊥X‖
26
THEOREM 2: limη→inf
β = β̂
η ∝ = 1‖Θ⊥β⊥X‖HOW WELL THE BASIS DESCRIBES X= ‖Θ⊥β⊥C‖η ∝ HOW POORLY THE BASIS DESCRIBES C
RECONSTRUCTIBILITY
Tuesday, September 7, 2010
-
X
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂C
X̂ = AX+ (1−A)C
Tuesday, September 7, 2010
-
X
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂Θ⊥β⊥CC
X̂ = AX+ (1−A)C
Tuesday, September 7, 2010
-
X
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂
Θ⊥β⊥X
Θ⊥β⊥CC
X̂ = AX+ (1−A)C
Tuesday, September 7, 2010
-
X̂ = AX+ (1−A)C
X
C
X̂
Θ
Θ⊥
R3F
GEOMETRY OF C AND X
βkβ2β1 …+ + +=
e
Trajectory Reconstruction
X
C
X̂Θ⊥β⊥C
Θ⊥β⊥X
Tuesday, September 7, 2010
-
WHAT DOES THIS TELL US?
• DE-CORRELATE CAMERA AND OBJECT MOTION
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
Tuesday, September 7, 2010
-
PARK ET AL. 2010IN PERSPECTIVE
• PROBLEMS SOLVED:• PERSPECTIVE RECONSTRUCTION• HANDLES MISSING DATA• LINEAR SOLVE (FAST, GLOBAL OPTIMUM)
• OPEN PROBLEMS:• HANDLING SMOOTHLY MOVING CAMERAS• AUTOMATIC COMPUTATION OF K• EXPLOITING DEPENDENCIES BETWEEN TRAJECTORIES• “PHYSICS-AWARE” ESTIMATION
Tuesday, September 7, 2010
-
http://www.cs.cmu.edu/~yaser/ECCV2010Tutorial.html [email protected]
37
COLLABORATORS
Ijaz AkhterHyun Soo Park
Eakta JainMoshe Mahler
Takaaki ShiratoriIain MatthewsJessica HodginsTakeo Kanade
Tuesday, September 7, 2010