k-means and gaussian mixture model 王养浩 2013 年 11 月 20 日
TRANSCRIPT
![Page 1: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/1.jpg)
SDP-MARCH-Talk
K-means and Gaussian Mixture Model
王养浩2013年 11月 20日
![Page 2: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/2.jpg)
Outline
• K-means• Gaussian Mixture Model• Expectation Maximum
![Page 3: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/3.jpg)
K-means
• Gather data points to a few cohesive ‘Clusters’
• Unsupervised Learning
![Page 4: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/4.jpg)
K-means
![Page 5: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/5.jpg)
K-means
![Page 6: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/6.jpg)
K-means
• Easy• Fast
• Euclidean distance?• K needs input?• Convergence?
![Page 7: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/7.jpg)
Determination of K
• Rule of Thumb:• Elbow Method• Cross Validation
2/nk
![Page 8: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/8.jpg)
K-means Convergence
• x(i) data points • μc(i) cluster centroids
• Coordinate descent
![Page 9: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/9.jpg)
Coordinate Descent
![Page 10: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/10.jpg)
K-means Convergence
• Local minimum– The optimization object is non-convex
![Page 11: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/11.jpg)
Gaussian Mixture Model
• Mixture of Gaussian distribution
![Page 12: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/12.jpg)
Gaussian Mixture Model
• Log likelihood
• Maximum likelihood – Expectation Maximum
![Page 13: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/13.jpg)
Expectation Maximum
![Page 14: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/14.jpg)
Expectation Maximum
• Jenson inquality
![Page 15: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/15.jpg)
Expectation Maximum
• Training set {}• Hidden variables {}• Parameter θ
![Page 16: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/16.jpg)
Expectation Maximum
• Construct lower bound
![Page 17: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/17.jpg)
Expectation Maximum
• Maximum lower bound– Coordinate Ascent on J
• Repeat until convergence –Maximum given fixed θ–Maximum θ given fixed
![Page 18: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/18.jpg)
Expectation Maximum
• Repeat until convergence
![Page 19: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/19.jpg)
Generalized Expectation Maximum
• Difficulty in M-step
![Page 20: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/20.jpg)
Summary
• K-means– Coordinate descent
• Gaussian Mixture Model– Expectation Maximum
• Expectation Maximum–MLE for models with latent variables– Generalized EM
![Page 21: K-means and Gaussian Mixture Model 王养浩 2013 年 11 月 20 日](https://reader033.vdocuments.us/reader033/viewer/2022061402/551b5f995503465c7e8b61e4/html5/thumbnails/21.jpg)
• Thanks!