ln l.agapito
TRANSCRIPT
![Page 1: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/1.jpg)
Dense Variational Reconstruction of Non-RigidSurfaces from Monocular Video
Ravi Garg Anastasios Roussos∗ Lourdes Agapito∗
Queen Mary, University of London∗Now at UCL
Before This Paper
1 / 17
![Page 2: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/2.jpg)
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
![Page 3: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/3.jpg)
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
![Page 4: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/4.jpg)
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
![Page 5: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/5.jpg)
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
![Page 6: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/6.jpg)
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.2 / 17
![Page 7: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/7.jpg)
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
Goal: dense 3D reconstruction for every frame.
Goal: dense 3D reconstruction for every frame.
NO additional sensors.
NO pre-trained shape models.
NO surface template.
2 / 17
![Page 8: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/8.jpg)
Dense Non Rigid Structure from Motion
Input: monocular sequence of non-rigid surface.
...
NRSfM: ill posed problem
Goal: dense 3D reconstruction for every frame.
Goal: dense 3D reconstruction for every frame.
NO additional sensors.
NO pre-trained shape models.
NO surface template.
2 / 17
![Page 9: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/9.jpg)
Traditional Sparse Non Rigid Structure from Motion
...
3 / 17
![Page 10: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/10.jpg)
Traditional Sparse Non Rigid Structure from Motion
...Feature Tracking
...
3 / 17
![Page 11: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/11.jpg)
Traditional Sparse Non Rigid Structure from Motion
...Feature Tracking
...
...3D ShapeInference
3 / 17
![Page 12: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/12.jpg)
Traditional Sparse Non Rigid Structure from Motion
Priors
...Feature Tracking
... +
...3D ShapeInference
3 / 17
![Page 13: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/13.jpg)
Low Rank Prior for NRSfM
Shape Space
(Bregler, Hertzmann, Biermann, Recovering non-rigid 3D shape from image streams CVPR’00.)
Bregler et al. CVPR’00, Brand CVPR’01, Xiao et al. IJCV’06, Torresani et al. PAMI’08, Akhter et al.CVPR’09, Bartoli et al. CVPR2008, Paladini et al. IJCV’12,Dai et al. CVPR’12...
4 / 17
![Page 14: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/14.jpg)
Low Rank Prior for NRSfM
Shape Space
(Bregler, Hertzmann, Biermann, Recovering non-rigid 3D shape from image streams CVPR’00.)
(Park et al. ECCV’10)
4 / 17
![Page 15: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/15.jpg)
Inspiration from Dense Rigid Reconstruction
(Newcombe, Lovegrove, Davison, DTAM: Dense Tracking and Mapping in Real-Time, ICCV’11)
5 / 17
![Page 16: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/16.jpg)
Inspiration from Dense Rigid Reconstruction
(Newcombe, Lovegrove, Davison, DTAM: Dense Tracking and Mapping in Real-Time, ICCV’11)
Key features
Variational approach.
Use of smoothness priors.
Per pixel reconstruction.
Scalable and GPU friendly algorithm.
5 / 17
![Page 17: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/17.jpg)
Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
6 / 17
![Page 18: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/18.jpg)
Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
We take the best of both worlds:
Low rank prior from sparse non rigid SfM.Variational framework from dense rigid SfM.
6 / 17
![Page 19: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/19.jpg)
Leap from sparse to dense NRSfM
Sparse
Dai et al. CVPR’12
Dense
This work
We take the best of both worlds:
Low rank prior from sparse non rigid SfM.Variational framework from dense rigid SfM.
Our contribution
First variational formulation to dense NRSfM.
Scalable algorithm which can be ported on GPU.
6 / 17
![Page 20: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/20.jpg)
Our Approach in a Nutshell
...
7 / 17
![Page 21: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/21.jpg)
Our Approach in a Nutshell
...
...
Step 1: DenseVideo Registration ...
7 / 17
![Page 22: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/22.jpg)
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
...
...
Step 1: DenseVideo Registration ...
7 / 17
![Page 23: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/23.jpg)
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
...
...
Step 1: DenseVideo Registration ...
Priors +
7 / 17
![Page 24: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/24.jpg)
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
...
...
Step 1: DenseVideo Registration ...
Low rank.Spatial smoothness.+
7 / 17
![Page 25: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/25.jpg)
Our Approach in a Nutshell
...
... Low rank.Spatial smoothness.
Step 1: DenseVideo Registration ...
+
Garg, Roussos, Agapito, A variational approach to video registration with subspace constraints, IJCV’13.7 / 17
![Page 26: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/26.jpg)
Our Approach in a Nutshell
Step 2: DenseShape Inference ...
... Low rank.Spatial smoothness.+
7 / 17
![Page 27: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/27.jpg)
Orthographic Projection Model
8 / 17
![Page 28: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/28.jpg)
Orthographic Projection Model
8 / 17
![Page 29: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/29.jpg)
Orthographic Projection Model
8 / 17
![Page 30: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/30.jpg)
Orthographic Projection Model
8 / 17
![Page 31: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/31.jpg)
Orthographic Projection Model
W = RS
8 / 17
![Page 32: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/32.jpg)
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
![Page 33: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/33.jpg)
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
![Page 34: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/34.jpg)
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
![Page 35: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/35.jpg)
Energy Minimisation Approach to NRSfM
Formulation of a single unified energy to estimate:
Orthographic projection matrices
3D shapes for all the frames
E(
R , S)
= λ Edata
(R,S
)+ Ereg
(S)
+ τ Etrace
(S)
reprojection error over all frames
spatial smoothness prior on 3D shapes
low rank prior on 3D shapes
9 / 17
![Page 36: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/36.jpg)
Reprojection ErrorE
`R,S
´= λEdata
`R,S
´+ Ereg
`S
´+ τEtrace
`S
´Edata (R,S) = ‖W − RS‖2F
10 / 17
![Page 37: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/37.jpg)
Spatial Smoothness PriorE
`R,S
´= λEdata
`R,S
´+ Ereg
`S
´+ τEtrace
`S
´Ereg
(S)
=∑
i
TV (Si)
−−−−−−→
Without regularisation With regularisation11 / 17
![Page 38: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/38.jpg)
Low Rank PriorE
`R,S
´= λEdata
`R,S
´+ Ereg
`S
´+ τEtrace
`S
´
Etrace
(S)
= ‖S‖∗ =∑
i
σi(S)
lies in−−−−→ span
K � F
Angst et al. ECCV’12, Dai et al. CVPR’12, Angst et al. ICCV’11, Dai et al. ECCV’10
12 / 17
![Page 39: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/39.jpg)
Minimisation of E(R,S
)
minR,Sλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
13 / 17
![Page 40: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/40.jpg)
Minimisation of E(R,S
)
minR,Sλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
Our Algorithm
Initialize R and S using rigid factorisation.
Minimize energy via alternation:
Step 1: Rotation estimation.Step 2: Shape estimation.
Efficient and highly parallelizable algorithm → GPU-friendly
13 / 17
![Page 41: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/41.jpg)
Minimisation of E(R,S
)
minRλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
Step 1: Rotation estimation
Robust estimation by using dense data.
Solved via Levenberg-Marquardt algorithm.
Rotations are parametrised as quaternions.
13 / 17
![Page 42: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/42.jpg)
Minimisation of E(R,S
)
minSλ ‖W − RS‖2F︸ ︷︷ ︸
Reprojectionerror
+∑i
TV (Si)︸ ︷︷ ︸Smoothness
prior
+ τ ‖S‖∗︸︷︷︸Low rank
prior
Step 2: Shape estimation
Convex sub-problem.
Optimisation via alternation between:
Per frame shape refinement: using primal dual algorithmEnforcing low rank: using soft impute algorithm.
13 / 17
![Page 43: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/43.jpg)
Results on real sequences
14 / 17
![Page 44: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/44.jpg)
Quantitative Evaluation
Average RMS 3D reconstruction errors.
Sequence TB MP Ours Ours(τ = 0)
Non-smooth rotations 4.50% 5.13% 2.60% 3.32%Smooth rotations 6.61% 5.81% 2.81% 3.89%
- TB: Akhter et al, Trajectory space: A dual representation for non-rigid structure from motion, PAMI’11.
- MP: Paladini et al, Optimal metric projections for deformable and articulated structure-from-motion,IJCV’12.
15 / 17
![Page 45: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/45.jpg)
Conclusions and future work
Conclusions:
First dense, template-free approach to Non-rigid Structurefrom Motion.Unified energy minimization for both rotation and shapeestimation.Combination of low-rank and spatial regularization prior.Using variational methods, we can do much more withmonocular sequences that one could expect
Future work:
Direct estimation from pixel intensitiesTowards real-time: online formulationOcclusion modelling
16 / 17
![Page 46: Ln l.agapito](https://reader034.vdocuments.us/reader034/viewer/2022042700/554e753bb4c90545698b4c85/html5/thumbnails/46.jpg)
Thank You for Your Attention!
For more details and data
Visit: www.eecs.qmul.ac.uk/~rgargOR Come to our poster!
Questions?
Authors thank
Chris Russell and Sara Vicente for valuable discussions.17 / 17