deformation @ tu

Computer Vision and Remote Sensing Technical University Berlin, Germany

Department of Electrical EngineeringIIT Roorkee, India

Using Morphable Face Model to Improve Stereo Reconstruction and Visualising the

Model on a Smartphone

HARDIK JAINUnder the Guidance of

Prof. Olaf HELLWICH

Prof. RS ANAND



PRESENTATION OUTLINE

Introduction

Motivation

Research Methodology

Results and Evaluation

Visualisation

Conclusion and Scope for Future Work

Further Reading

1 of 25



INTRODUCTION

Face Reconstruction

Multiple Image ReconstructionVisual structure from motion

Stereo Reconstruction

Single Image Reconstruction3D Morphable Model

2 of 25



3D Morphable Model

Statistical model of face meshes which are in dense correspondence.

Principal component analysis (PCA) is performed on these set of 𝑀 meshes.

Mean shape ҧ𝑠, 𝑀 − 1 Principal Components 𝑆𝑖 and 𝜎𝑠,𝑖2

eigen values

Shape parameter vector cs = 𝛼1, … , 𝛼𝑀−1𝑇

iisi

M

i

model SsS 2

,

1

1=

=

INTRODUCTI

ONSingle Image Reconstruction

Morphable Model

3 of 25



MOTIVATION Stereo Reconstruction

Stereo Model

Stereo Model

High Quality

Face Scan

Stereo Model and High Quality Scan Cloud

Compare

4 of 25



Face Model

MOTIVATION Single Image Reconstruction

Face Image Face Model

Face Model Cloud Compare

5 of 25



MOTIVATION

Single Image

Reconstruction

Stereo Model

Deformed Face Model

Shape Information

Texture & Smoothness

Deformation

6 of 25



RESEARCH METHODOLOGY

MorphableModel

Stereo Pair Image

Deformed Face Model

Single ImageLandmarking

Pose EstimationShape Fitting

Texture Extraction

Global & Local Deformation

Stereo Reconstruction

Face Model

7 of 25

Method Overview



Landmarking Annotation

To find 𝐅 = 𝑥1 𝑦1, … , 𝑥𝑛 𝑦𝑛𝑇 ∈ ℝ2𝑛

Obtain Face Bounding Box

Cascade based Regression Method Initial estimate is centred to the Bounding box

Regressor rt 𝐼, F(t)

F(t+1) = F(t) + rt 𝐼, F(t)

RESEARCH

METHODOLOGYSingle Image Reconstruction

Landmark Annotated Face Image

8 of 25



Pose Estimation

Camera Orientation Matrix, 𝐏

𝐱𝒊 = 𝐏𝐗𝒊 ,where 𝐱𝒊∈ ℝ2 and 𝐗𝒊∈ ℝ3

Affine Camera Model

Gold Standard Algorithm of Hartley & Zisserman

RESEARCH


9 of 25

2D Landmark Points

3D Points on MM



Shape Fitting

Estimation of shape Parameter 𝐜s

Probabilistic Approach

Minimise 𝐸 = σ𝑖=13𝑁 𝑦 𝑚𝑜𝑑𝑒𝑙2𝐷,𝑖 −𝑦𝑖

2

2 𝜎 2𝐷,𝑖2 + 𝐜s

2

Face Image Face Shape Model

RESEARCH


iisi

M

i

model SsS 2

,

1

1=

=

10 of 25



Texture Extraction

Obtained from Face Image

Isomap Algorithm

Retaining Geodesic Distance

Texture Map

RESEARCH


Face Image

11 of 25



STEREO RECONSTRUCTION

Obtain stereo pair images

Camera Calibration

Keypoint Detection

Triangulation

Dense Reconstruction

RESEARCH

METHODOLOGY

Stereo Model

12 of 25



DEFORMATION

Motivation

Surface Registration

Approach

Two-step deformation

Global and Local transformation

𝒗𝑗 = Φ𝑙𝑜𝑐𝑎𝑙 ∘ Φ𝑔𝑙𝑜𝑏𝑎𝑙 (𝒗𝑗)

RESEARCH

METHODOLOGY

13 of 25



Global Deformation

Using few control points

Radial Basis Function 𝜙 𝑥, 𝑋𝑐 = 𝜙 𝑥 − 𝑋𝑐

Weighted Combination of RBF

𝑔 𝑥 =

𝑘=1

𝑁

𝜆𝑐 𝜙 𝑥 − 𝑋𝑐

Gaussian Kernel 𝜙𝑖,𝑐 𝑥 = 𝑒−

𝑥𝑖− 𝑋𝑐2

2𝜎2

RESEARCH

METHODOLOGYDeformation

14 of 25



𝜆𝑐 = 𝜙−1𝑔 applied on Sourcepartial

𝜆𝑟,1⋮

𝜆𝑟,𝑁

=

𝜙1,1 ⋯ 𝜙1,𝑁⋮ ⋱ ⋮

𝜙𝑉,1 ⋯ 𝜙𝑉,𝑁

−1 𝑔𝑟,1⋮

𝑔𝑟,𝑉, for r = (x,y,z)

Face unspecific results for N=25

Few 100 milliseconds on intel quad core computer

RESEARCH

METHODOLOGYDeformation Global Deformation

Weight Coefficients (𝜆𝑐)

15 of 25



Local Deformation

Smoothen the overshoots of RBF

Influence of nearby neighbors

Non-rigid transformation by k nearest neighbouring vertices

Procrustes Analysis

Affine transformation (𝐁𝑖 and 𝐭𝑖)

RESEARCH

METHODOLOGYDeformation

16 of 25



Φlocal 𝑣 = w0 𝑣 𝐁0𝑣 + 𝐭0 + σi=1𝐾 wi 𝑣 𝐁i𝑣 + 𝐭i

wi 𝑣 =1

𝐾

d− 𝑣𝑖− 𝑣

d, 𝑑 = σi=1

𝐾 𝑣𝑖 − 𝑣

w0(𝑣) =1

𝐾

𝐾 = 12

Requires Few seconds

RESEARCH

METHODOLOGYDEFORMATION

Local Deformation

1

5

4

2

3

6

17 of 25



DEFORMED FACE MODEL

RESULTS AND

EVALUATION

Face Image Face Model Deformed Face Model

18 of 25



EVALUATION

RESULTS AND

EVALUATION

19 of 25

3.5532 3.3073 2.8738



EVALUATION

RESULTS AND

EVALUATION

20 of 25

4.2734 2.9985 2.0133



EVALUATION

RESULTS AND

EVALUATION

21 of 25

2.9982 2.5212 2.174



VISUALISATION

Virtual Reality

Using Smartphone and Cardboard Viewer

Android phones with support of OpenGL ES 3.1

Implemented on SDK provided by Google

22 of 25



Smartphone View

VISUALISATION

23 of 25



CONCLUSION AND SCOPE FOR FUTURE WORK

Improved reconstruction after information fusion

Technique Could be used for various other objects

Cheap alternative visualisation platform

Smartphone visualisation Environment Improvement

24 of 25



FURTHER READING V. Blanz and T. Vetter, “A morphable model for the synthesis of 3D faces,” in Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 1999, pp. 187–194.

P. Huber, G. Hu, R. Tena, P. Mortazavian, W. P. Koppen, W. Christmas, M. Ratsch, and J. Kittler, “A multiresolution 3D morphable face model and fitting framework,” in 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, February 2016.

P. Huber, Z.-H. Feng, W. Christmas, J. Kittler, and M. Rätsch, “Fitting 3D morphable models using local features,” arXiv preprint arXiv:1503.02330, 2015.

V. Kazemi and J. Sullivan, “One millisecond face alignment with an ensemble of regression trees,” in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014, pp. 1867– 1874.

R. W. Sumner, J. Schmid, and M. Pauly, “Embedded deformation for shape manipulation,” ACM Transactions on Graphics (TOG), vol. 26, no. 3, p. 80, 2007.

“Cardboard.” https://developers.google.com/cardboard/overview, 2016.

25 of 25

deformation @ tu

Documents