direct fitting of gaussian mixture modelsmesh registration with p2d method 1. apply a random...

Direct Fitting of Gaussian Mixture Models

Leonid Keselman, Martial Hebert

Robotics InstituteCarnegie Mellon University

May 29, 2019

https://github.com/leonidk/direct_gmm1

Representations of 3D data

2

Point Cloud Point Cloud+ Normals

Triangular Mesh

Nearest Neighbor Plane Fit Screened Poisson Surface Reconstruction

Gaussian Mixture Models for 3D Shapes

3

GMM fit to object surface

Benefits• Closed-form expression• Can represent contiguous surfaces• Easy to build from noisy data• Sparse

Gaussian Mixture Model (GMM)

4

! " = $%&'

()* +("; .*, Σ*) $

*)* = 1)* ≥ 0

Σ* is symmetric, positive-semidefinite

Gaussian Mixtures as a shape representation

W. Tabib, C. O’Meadhra, N. MichaelIEEE R-AL (2018)

B. Eckart, K. Kim, A. Troccoli, A. Kelly, J. Kautz. CVPR (2016)

B. Eckart, K. Kim, J. Kautz. ECCV (2018)

5

Efficient Representation Mesh Registration Frame Registration

Fitting a Gaussian Mixture Model

1. Obtain 3D Point Cloud2. Select Initial Parameters3. Iterate Expectation & Maximization

i. E-Step: Each point gets a likelihoodii. M-Step: Each mixture gets parameters

6

7

The E-Step (Given GMM parameters)

8

!"# =1&#'" ((*#; ,", Σ")

&# =01'1 ((*#; ,1, Σ1)

Normalization constant for point j

Affiliation between point j & mixture i

The M-Step (Given point-mixture weights)

9

!" = $%&'

($)&'

*+)%log /) 0(2%; 4), Σ))

8!"8/)

= 0

mixturespoints

8!"84)

= 0 8!"8Σ)

= 0

lower-bound loss

To get new parameters: takes derivatives, set equal to zero, and solve

4) =1;)$

%<)%2%/) =

;)=

Σ) =1;)$

%<)%(2%−4))(2%−4))?

<)% = +)%

;) =$%<)%

10

Geometric Objects in a Probability Distribution

11

Known curve in a given 2D probability distribution

Geometric Objects in a Probability Distribution

12

Consider sampling N points from this curve

ℓ curve ≅()*+

,-(/))

Geometric Objects in a Probability Distribution

13

Take a geometric mean to account for sample number

ℓ curve ≅ ()*+

,-(/))

+,

Geometric Objects in a Probability Distribution

14

The curve will be the value in the limit

ℓ curve ≅ ()*+

,-(/))

+,

ℓ curve = lim,→6 ()*+

,-(/))

+,

= lim,→6exp log ()*+

,-(/))

+,

= lim,→6exp1< =

)*+

,log(-(/)))

Geometric Objects in a Probability Distribution

15

ℓ curve ≅ ()*+

,-(/))

+,

ℓ curve = lim,→6 ()*+

,-(/))

+,

= lim,→6exp log ()*+

,-(/))

+,

= lim,→6exp1< =

)*+

,log(-(/))) = exp > log(-(/)) ?/

Geometric Objects in a Probability Distribution

16

!= exp & log(+(,)) .,

1. If +(,) = 0 on curve, then L= 02. Invariant to reparameterization

17

!" Area of each triangle#" Centroid of each triangle$", &", '" Triangle vertices

The E-Step (Given GMM parameters)

18

!"# =1&#'" ((*#; ,", Σ")

&# =01'1 ((*#; ,1, Σ1)

Normalization constant for point j

Affiliation between point j & mixture i

2# Area of each triangle,# Centroid of each triangle3#, 4#, &# Triangle vertices

The New E-Step (Given GMM parameters)

19

!"# =1&#'"(# )(+#; +", Σ")

&# =01'1(1)(+#; +1, Σ1)

Normalization constant for object j

Affiliation between object j & mixture i

(# Area of each triangle+# Centroid of each triangle2#, 3#, &# Triangle vertices

Taylor Approximation(2 terms)

The M-Step (Given point-mixture weights)

20

!" =1%"&

'("')'

*" =%"+

Σ" =1%"&

'("'()'−!")()'−!")0

("' = 1"'

%" =&'("'

The New M-Step (Given point-mixture weights)

21

!" =1%"&

'("')'

*" =%"+

Σ" =1%"&

'("' ()'−!")()'−!")0 + Σ'

("' = 2'3"'

%" =&'("'

2' Area of each triangle!' Centroid of each triangle4', 6', 7' Triangle vertices

Σ' =112 4'4'0 + 6'6'0 + 7'7'0 − 3 !'!'0

What is Σ"?

22

23

ResultsDid all that math actually help us fit better/faster GMMs?

24

25

Using different inputs

classic algorithm• Vertices of the mesh• Triangle centroids

our method• Approximate (E only)• Exact (E + M steps)

Measure the likelihood of a high-density point cloud (higher is better)

Evaluate across a wide range of mixtures(6 to 300)

Full E+M method works in all cases

26

Stable under even random initialization!

27

ApplicationsAre these models actually more useful?

28

Mesh Registration (P2D)

Method1. Apply a random rotation + translation to the point cloud2. Find transformation to maximize the likelihood of the points

• Perform P2D with GMMs fit toi. mesh verticesii. mesh triangles

29

Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” ECCV (2018)

Mesh-based GMMs are more accurate

30

Across multiple models

31

0

50

100

150

Armadillo Bunny Dragon Happy Lucy

Rotation Error(% of ICP)

points mesh

0

100

200

Armadillo Bunny Dragon Happy Lucy

Translation Error(% of ICP)

points mesh

Frame Registration (D2D)

Method1. Use a sequence from an RGBD Sensor

• 2,500 frame TUM sequence from a Microsoft Kinect

2. Pairwise registration between t & t-1 frames• Optimize the D2D L2 distance• Build GMMs using square pixels as the geometric object

32

W. Tabib, C. O’Meadhra, N. Michael.“On-Manifold GMM Registration” IEEE R-AL (2018)

Representing points using pixel squares

33

D2D Registration Results

34

Compared to standard GMM• 2.4% improvement in RMSE• 22% faster D2D convergence

Questions?

35

! = exp & log(+(,)) .,

The End!

36

Extra Slides

37

How to fit a Gaussian Mixture Model?

1. Obtain any collection of objects2. Perform Expectation + Maximization

i. E-Step: Each point gets a likelihoodii. M-Step: Each mixture gets new parameters

38

Extension to arbitrary primitives

39Vasconcelos, Lippman. "Learning mixture hierarchies.” Advances in Neural Information Processing Systems (1999)

Approximation

40

area-weighted geometric mean using the primitive’s centroids

Product Integral Formulation

• Product integrals provide a resampling-invariant loss function• Given S samples, of M primitives, with N mixture components

• This can be evaluated in the limit of samples (with a geometric mean)

41

42

43

44

For GMMs we will use the lower bound

45

P2D Registration Results

46

Model Rotation Error(% of ICP)

Translation Error(% of ICP)

points mesh points meshArmadillo 127 37 161 33Bunny 50 28 41 17Dragon 68 25 40 19Happy 101 27 85 27Lucy 95 23 122 35

Mesh Registration with P2D

Method1. Apply a random rotation + translation to the point cloud2. Point-to-Distribution (P2D) registration of point cloud to GMM

• Perform tests with GMMs fit toi. mesh verticesii. mesh triangles

• Optimize the GMM likelihood with rigid body transformation (q & t)• BFGS Optimization using numerical gradients, starting from identity

47

Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” ECCV (2018)

Acknowledgements

• Martial Hebert• Robotics Institute• Reviewers• Funding?

48

Representing points using pixel squares

49