parametric density estimation using gaussian mixture models
DESCRIPTION
Based on tutorials by Jeff A. Bilmes and by Ludwig SchwardtSlides by Pardis NoorzadTRANSCRIPT
![Page 1: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/1.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Parametric Density Estimation using
Gaussian Mixture Models
An Application of the EM Algorithm
Based on tutorials by Je� A. Bilmes and by Ludwig Schwardt
Amirkabir University of Technology � Bahman 12, 1389
![Page 2: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/2.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 3: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/3.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 4: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/4.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 5: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/5.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 6: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/6.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 7: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/7.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 8: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/8.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
What does EM do?
iterative maximization of the likelihood function
when data has missing values or
when maximization of the likelihood is di�cult
data augmentation
X is incomplete data
Z = (X;Y) is the complete data set
![Page 9: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/9.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
What does EM do?
iterative maximization of the likelihood function
when data has missing values or
when maximization of the likelihood is di�cult
data augmentation
X is incomplete data
Z = (X;Y) is the complete data set
![Page 10: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/10.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
What does EM do?
iterative maximization of the likelihood function
when data has missing values or
when maximization of the likelihood is di�cult
data augmentation
X is incomplete data
Z = (X;Y) is the complete data set
![Page 11: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/11.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
What does EM do?
iterative maximization of the likelihood function
when data has missing values or
when maximization of the likelihood is di�cult
data augmentation
X is incomplete data
Z = (X;Y) is the complete data set
![Page 12: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/12.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
What does EM do?
iterative maximization of the likelihood function
when data has missing values or
when maximization of the likelihood is di�cult
data augmentation
X is incomplete data
Z = (X;Y) is the complete data set
![Page 13: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/13.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
What does EM do?
iterative maximization of the likelihood function
when data has missing values or
when maximization of the likelihood is di�cult
data augmentation
X is incomplete data
Z = (X;Y) is the complete data set
![Page 14: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/14.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
What does EM do?
iterative maximization of the likelihood function
when data has missing values or
when maximization of the likelihood is di�cult
data augmentation
X is incomplete data
Z = (X;Y) is the complete data set
![Page 15: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/15.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
EM and MLE
maximum likelihood estimation
estimates density parameters
=) EM app: parametric density estimation
when density is a mixture of Gaussians
![Page 16: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/16.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
EM and MLE
maximum likelihood estimation
estimates density parameters
=) EM app: parametric density estimation
when density is a mixture of Gaussians
![Page 17: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/17.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
EM and MLE
maximum likelihood estimation
estimates density parameters
=) EM app: parametric density estimation
when density is a mixture of Gaussians
![Page 18: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/18.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
EM and MLE
maximum likelihood estimation
estimates density parameters
=) EM app: parametric density estimation
when density is a mixture of Gaussians
![Page 19: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/19.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Introduction
EM and MLE
maximum likelihood estimation
estimates density parameters
=) EM app: parametric density estimation
when density is a mixture of Gaussians
![Page 20: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/20.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 21: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/21.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The density estimation problem
De�nition
Our parametric density estimation problem:
X = fxigNi=1, xi 2 RD
f(xijΘ), i.i.d. assumption
f(xijΘ) =PK
k=1 �kp(xij�k)
K components (given)
Θ = (�1; : : : ; �K ; �1; : : : ; �K)
![Page 22: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/22.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The density estimation problem
De�nition
Our parametric density estimation problem:
X = fxigNi=1, xi 2 RD
f(xijΘ), i.i.d. assumption
f(xijΘ) =PK
k=1 �kp(xij�k)
K components (given)
Θ = (�1; : : : ; �K ; �1; : : : ; �K)
![Page 23: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/23.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The density estimation problem
De�nition
Our parametric density estimation problem:
X = fxigNi=1, xi 2 RD
f(xijΘ), i.i.d. assumption
f(xijΘ) =PK
k=1 �kp(xij�k)
K components (given)
Θ = (�1; : : : ; �K ; �1; : : : ; �K)
![Page 24: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/24.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The density estimation problem
De�nition
Our parametric density estimation problem:
X = fxigNi=1, xi 2 RD
f(xijΘ), i.i.d. assumption
f(xijΘ) =PK
k=1 �kp(xij�k)
K components (given)
Θ = (�1; : : : ; �K ; �1; : : : ; �K)
![Page 25: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/25.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The density estimation problem
De�nition
Our parametric density estimation problem:
X = fxigNi=1, xi 2 RD
f(xijΘ), i.i.d. assumption
f(xijΘ) =PK
k=1 �kp(xij�k)
K components (given)
Θ = (�1; : : : ; �K ; �1; : : : ; �K)
![Page 26: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/26.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The density estimation problem
De�nition
Our parametric density estimation problem:
X = fxigNi=1, xi 2 RD
f(xijΘ), i.i.d. assumption
f(xijΘ) =PK
k=1 �kp(xij�k)
K components (given)
Θ = (�1; : : : ; �K ; �1; : : : ; �K)
![Page 27: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/27.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
Constraint on mixing probabilities �iSum to unity
ZRD
p(xij�k)dxi = 1
1 =
ZRD
f(xijΘ)dxi
=
ZRD
KXk=1
�kp(xij�k)dxi
=KXk=1
�k:
![Page 28: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/28.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
Constraint on mixing probabilities �iSum to unity
ZRD
p(xij�k)dxi = 1
1 =
ZRD
f(xijΘ)dxi
=
ZRD
KXk=1
�kp(xij�k)dxi
=KXk=1
�k:
![Page 29: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/29.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The maximum likelihood problem
De�nition
Our MLE problem:
p(XjΘ) =QNi=1 p(xijΘ) = L(ΘjX)
Θ� = argmaxΘ L(ΘjX)
the log-likelihood: log(L(ΘjX))
log(L(ΘjX)) =PN
i=1 log�PK
k=1 �kp(xi; �k)�
di�cult to optimize b/c it contains log of sum
![Page 30: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/30.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The maximum likelihood problem
De�nition
Our MLE problem:
p(XjΘ) =QNi=1 p(xijΘ) = L(ΘjX)
Θ� = argmaxΘ L(ΘjX)
the log-likelihood: log(L(ΘjX))
log(L(ΘjX)) =PN
i=1 log�PK
k=1 �kp(xi; �k)�
di�cult to optimize b/c it contains log of sum
![Page 31: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/31.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The maximum likelihood problem
De�nition
Our MLE problem:
p(XjΘ) =QNi=1 p(xijΘ) = L(ΘjX)
Θ� = argmaxΘ L(ΘjX)
the log-likelihood: log(L(ΘjX))
log(L(ΘjX)) =PN
i=1 log�PK
k=1 �kp(xi; �k)�
di�cult to optimize b/c it contains log of sum
![Page 32: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/32.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The maximum likelihood problem
De�nition
Our MLE problem:
p(XjΘ) =QNi=1 p(xijΘ) = L(ΘjX)
Θ� = argmaxΘ L(ΘjX)
the log-likelihood: log(L(ΘjX))
log(L(ΘjX)) =PN
i=1 log�PK
k=1 �kp(xi; �k)�
di�cult to optimize b/c it contains log of sum
![Page 33: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/33.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The maximum likelihood problem
De�nition
Our MLE problem:
p(XjΘ) =QNi=1 p(xijΘ) = L(ΘjX)
Θ� = argmaxΘ L(ΘjX)
the log-likelihood: log(L(ΘjX))
log(L(ΘjX)) =PN
i=1 log�PK
k=1 �kp(xi; �k)�
di�cult to optimize b/c it contains log of sum
![Page 34: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/34.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The maximum likelihood problem
De�nition
Our MLE problem:
p(XjΘ) =QNi=1 p(xijΘ) = L(ΘjX)
Θ� = argmaxΘ L(ΘjX)
the log-likelihood: log(L(ΘjX))
log(L(ΘjX)) =PN
i=1 log�PK
k=1 �kp(xi; �k)�
di�cult to optimize b/c it contains log of sum
![Page 35: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/35.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulation
De�nition
Our EM problem:
Z = (X;Y)
L(ΘjZ) = L(ΘjX;Y) = p(X;YjΘ)
yi 2 1 : : :K
yi = k if the ith sample was generated by the kth component
Y is a random vector
log(L(ΘjX;Y)) =PN
i=1 log(�yip(xij�yi))
![Page 36: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/36.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulation
De�nition
Our EM problem:
Z = (X;Y)
L(ΘjZ) = L(ΘjX;Y) = p(X;YjΘ)
yi 2 1 : : :K
yi = k if the ith sample was generated by the kth component
Y is a random vector
log(L(ΘjX;Y)) =PN
i=1 log(�yip(xij�yi))
![Page 37: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/37.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulation
De�nition
Our EM problem:
Z = (X;Y)
L(ΘjZ) = L(ΘjX;Y) = p(X;YjΘ)
yi 2 1 : : :K
yi = k if the ith sample was generated by the kth component
Y is a random vector
log(L(ΘjX;Y)) =PN
i=1 log(�yip(xij�yi))
![Page 38: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/38.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulation
De�nition
Our EM problem:
Z = (X;Y)
L(ΘjZ) = L(ΘjX;Y) = p(X;YjΘ)
yi 2 1 : : :K
yi = k if the ith sample was generated by the kth component
Y is a random vector
log(L(ΘjX;Y)) =PN
i=1 log(�yip(xij�yi))
![Page 39: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/39.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulation
De�nition
Our EM problem:
Z = (X;Y)
L(ΘjZ) = L(ΘjX;Y) = p(X;YjΘ)
yi 2 1 : : :K
yi = k if the ith sample was generated by the kth component
Y is a random vector
log(L(ΘjX;Y)) =PN
i=1 log(�yip(xij�yi))
![Page 40: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/40.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulation
De�nition
Our EM problem:
Z = (X;Y)
L(ΘjZ) = L(ΘjX;Y) = p(X;YjΘ)
yi 2 1 : : :K
yi = k if the ith sample was generated by the kth component
Y is a random vector
log(L(ΘjX;Y)) =PN
i=1 log(�yip(xij�yi))
![Page 41: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/41.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulation
De�nition
Our EM problem:
Z = (X;Y)
L(ΘjZ) = L(ΘjX;Y) = p(X;YjΘ)
yi 2 1 : : :K
yi = k if the ith sample was generated by the kth component
Y is a random vector
log(L(ΘjX;Y)) =PN
i=1 log(�yip(xij�yi))
![Page 42: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/42.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationContinued
De�nition
The E- and M-steps with the auxiliary function Q:
E-step: Q(Θ;Θ(i�1)) = E[log(p(X;YjΘ))jX;Θ(i�1)]
M-step: Θ(i) = argmaxΘQ(Θ;Θ(i�1))
![Page 43: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/43.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationContinued
De�nition
The E- and M-steps with the auxiliary function Q:
E-step: Q(Θ;Θ(i�1)) = E[log(p(X;YjΘ))jX;Θ(i�1)]
M-step: Θ(i) = argmaxΘQ(Θ;Θ(i�1))
![Page 44: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/44.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationContinued
De�nition
The E- and M-steps with the auxiliary function Q:
E-step: Q(Θ;Θ(i�1)) = E[log(p(X;YjΘ))jX;Θ(i�1)]
M-step: Θ(i) = argmaxΘQ(Θ;Θ(i�1))
![Page 45: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/45.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationThe Q function
Q(Θ;Θ(t)) = E[log(p(X;YjΘ))jX;Θ(t)]
=KXk=1
NXi=1
log(�kp(xij�k))p(kjxi;Θ(t))
=KXk=1
NXi=1
log(�k)p(kjxi;Θ(t))
+KXk=1
NXi=1
log(p(xij�k))p(kjxi;Θ(t))
![Page 46: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/46.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationSolving for �k
We use the Lagrange multiplier to enforceP
k �k = 1.
@
@�k[Xk
Xi
log(�k)p(kjxi;Θ(t)) + �(Xk
�k � 1)] = 0
Xi
1
�kp(kjxi;Θ(t)) + � = 0
Summing both sides over k, we get � = �N .
�k =1
N
Xi
p(kjxi;Θ(t)):
![Page 47: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/47.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationSolving for �k
We use the Lagrange multiplier to enforceP
k �k = 1.
@
@�k[Xk
Xi
log(�k)p(kjxi;Θ(t)) + �(Xk
�k � 1)] = 0
Xi
1
�kp(kjxi;Θ(t)) + � = 0
Summing both sides over k, we get � = �N .
�k =1
N
Xi
p(kjxi;Θ(t)):
![Page 48: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/48.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationSolving for �k
We use the Lagrange multiplier to enforceP
k �k = 1.
@
@�k[Xk
Xi
log(�k)p(kjxi;Θ(t)) + �(Xk
�k � 1)] = 0
Xi
1
�kp(kjxi;Θ(t)) + � = 0
Summing both sides over k, we get � = �N .
�k =1
N
Xi
p(kjxi;Θ(t)):
![Page 49: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/49.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationSolving for �k
We use the Lagrange multiplier to enforceP
k �k = 1.
@
@�k[Xk
Xi
log(�k)p(kjxi;Θ(t)) + �(Xk
�k � 1)] = 0
Xi
1
�kp(kjxi;Θ(t)) + � = 0
Summing both sides over k, we get � = �N .
�k =1
N
Xi
p(kjxi;Θ(t)):
![Page 50: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/50.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Density Estimation
The EM formulationSolving for �k
We use the Lagrange multiplier to enforceP
k �k = 1.
@
@�k[Xk
Xi
log(�k)p(kjxi;Θ(t)) + �(Xk
�k � 1)] = 0
Xi
1
�kp(kjxi;Θ(t)) + � = 0
Summing both sides over k, we get � = �N .
�k =1
N
Xi
p(kjxi;Θ(t)):
![Page 51: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/51.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 52: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/52.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
GMMSolving for mk and Σk
Gaussian components
p(xijxk;Σk) = (2�)�D
2 jΣkj�
12 exp(�1
2(xi �mk)T�1
k (xi �mk))
mk =
Pi xip(kjxi;Θ
(t))Pi p(kjxi;Θ(t))
Σk =p(kjxi;Θ(t))(xi � mk)(xi � mk)
T
p(kjxi;Θ(t))
![Page 53: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/53.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
GMMSolving for mk and Σk
Gaussian components
p(xijxk;Σk) = (2�)�D
2 jΣkj�
12 exp(�1
2(xi �mk)T�1
k (xi �mk))
mk =
Pi xip(kjxi;Θ
(t))Pi p(kjxi;Θ(t))
Σk =p(kjxi;Θ(t))(xi � mk)(xi � mk)
T
p(kjxi;Θ(t))
![Page 54: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/54.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
Choice of ΣkFull covariance
Fits data best, but costly in high-dimensional space.
Figure: full covariance
![Page 55: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/55.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
Choice of ΣkDiagonal covariance
A tradeo� between cost and quality.
Figure: diagonal covariance
![Page 56: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/56.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
Choice of ΣkSpherical covariance, Σk = �2
kI
Needs many components to cover data.
Figure: spherical covariance
Decision should be based on the size of training data set
so that all parameters may be tuned
![Page 57: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/57.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
Choice of ΣkSpherical covariance, Σk = �2
kI
Needs many components to cover data.
Figure: spherical covariance
Decision should be based on the size of training data set
so that all parameters may be tuned
![Page 58: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/58.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Gaussian Mixture Model
Choice of ΣkSpherical covariance, Σk = �2
kI
Needs many components to cover data.
Figure: spherical covariance
Decision should be based on the size of training data set
so that all parameters may be tuned
![Page 59: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/59.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Some Results
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 60: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/60.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Some Results
Data
data generated from a mixture of three bivariate Gaussians
![Page 61: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/61.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Some Results
PDFs
estimated pdf contours (after 43 iterations)
![Page 62: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/62.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Some Results
Clusters
xi assigned to the cluster k corresponding to the highest p(kjxi;Θ)
![Page 63: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/63.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
Outline
1 Introduction
2 Density Estimation
3 Gaussian Mixture Model
4 Some Results
5 Other Applications of EM
![Page 64: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/64.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 65: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/65.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 66: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/66.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 67: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/67.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 68: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/68.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 69: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/69.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 70: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/70.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 71: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/71.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 72: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/72.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
A problem with missing dataLife-testing experiment
lifetime of light bulbs follows an exponential distribution with
mean �
two separate experiments
N bulbs tested until failed
failure times recorded as x1; : : : ; xN
M bulbs tested
failure times not recorded
only number of bulbs r, that failed at time t
missing data are failure times u1; : : : ; uM
![Page 73: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/73.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
EM formulationLife-testing experiment
log(L(�jx; u)) = �N(log � + �x=�)�PM
i (log � + ui=�)
expected value for bulb still burning: t+ �
one that burned out: � � te�t=�(k)
1�e�t=�(k)= � � th(k)
E-step:Q(�; �(k)) = �(N +M) log �
�1
�(N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
M-step: 1N+M (N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
![Page 74: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/74.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
EM formulationLife-testing experiment
log(L(�jx; u)) = �N(log � + �x=�)�PM
i (log � + ui=�)
expected value for bulb still burning: t+ �
one that burned out: � � te�t=�(k)
1�e�t=�(k)= � � th(k)
E-step:Q(�; �(k)) = �(N +M) log �
�1
�(N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
M-step: 1N+M (N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
![Page 75: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/75.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
EM formulationLife-testing experiment
log(L(�jx; u)) = �N(log � + �x=�)�PM
i (log � + ui=�)
expected value for bulb still burning: t+ �
one that burned out: � � te�t=�(k)
1�e�t=�(k)= � � th(k)
E-step:Q(�; �(k)) = �(N +M) log �
�1
�(N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
M-step: 1N+M (N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
![Page 76: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/76.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
EM formulationLife-testing experiment
log(L(�jx; u)) = �N(log � + �x=�)�PM
i (log � + ui=�)
expected value for bulb still burning: t+ �
one that burned out: � � te�t=�(k)
1�e�t=�(k)= � � th(k)
E-step:Q(�; �(k)) = �(N +M) log �
�1
�(N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
M-step: 1N+M (N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
![Page 77: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/77.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
EM formulationLife-testing experiment
log(L(�jx; u)) = �N(log � + �x=�)�PM
i (log � + ui=�)
expected value for bulb still burning: t+ �
one that burned out: � � te�t=�(k)
1�e�t=�(k)= � � th(k)
E-step:Q(�; �(k)) = �(N +M) log �
�1
�(N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
M-step: 1N+M (N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
![Page 78: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/78.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Other Applications of EM
EM formulationLife-testing experiment
log(L(�jx; u)) = �N(log � + �x=�)�PM
i (log � + ui=�)
expected value for bulb still burning: t+ �
one that burned out: � � te�t=�(k)
1�e�t=�(k)= � � th(k)
E-step:Q(�; �(k)) = �(N +M) log �
�1
�(N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
M-step: 1N+M (N �x+ (M � r)(t+ �(k)) + r(�(k) � th(k)))
![Page 79: Parametric Density Estimation using Gaussian Mixture Models](https://reader033.vdocuments.us/reader033/viewer/2022052412/559059401a28ab604b8b47a2/html5/thumbnails/79.jpg)
Parametric Density Estimation using Gaussian Mixture Models
Thank you for your attention. Any questions?