deep adversarial gaussian mixture auto-encoder …harchaoui.org/warith/dac_ppt.pdfoptimization...
TRANSCRIPT
![Page 1: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/1.jpg)
Deep Adversarial
Gaussian Mixture
Auto-Encoder
for Clustering
Warith HARCHAOUI Pierre-Alexandre MATTEI
Charles BOUVEYRON
Université Paris Descartes � MAP5
Oscaro.com � Research & Development
February 2017
1/17
![Page 2: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/2.jpg)
Clustering
Clustering is grouping similar objects together!
2/17
![Page 3: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/3.jpg)
Thesis
Representation Learning and Clustering operate a symbiosis
3/17
![Page 4: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/4.jpg)
Gaussian Mixture Model
I Density Estimation applied to Clustering for Kmodes/clusters
I Linear complexity suitable for Large Scale Problems
4/17
![Page 5: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/5.jpg)
Learning Representations
I Successful in a supervised context (Kernel SVM)
I Successful in an unsupervised context (Spectral Clustering)
5/17
![Page 6: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/6.jpg)
Auto-Encoder
An auto-encoder is a neural network that consists of:
I an Encoder: E : RD → Rd (compression)
I a Decoder: D : Rd → RD (decompression)
D >> d
D(E(x)) ' x
6/17
![Page 7: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/7.jpg)
Optimization Scheme
Input Space
Code Space
Gaussian Clusters (π, µ, Σ)
Encoder Decoder
GMM Discriminator
Figure: Global Optimization Scheme for DAC
7/17
![Page 8: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/8.jpg)
Adversarial Auto-Encoder
An adversarial auto-encoder is a neural network that consists of:
I an Encoder: E : RD → Rd (compression)
I a Decoder: D : Rd → RD (decompression)
I a Prior: P : Rd → R and∫Rd P = 1 associated with a random
generator of distribution PI a Discriminator: A : Rd → [0, 1] ⊂ R that distinguishes fake
data from the random generator and real data from the
encoder
8/17
![Page 9: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/9.jpg)
Optimizations
3 lines objectives:
I The encoder and decoder try to minimize the reconstruction
loss
I The discriminator tries to distinguish fake codes (from the
random generator associated with the prior) and real codes
(from the encoder)
I The encoder also tries to fool the discriminator (opposite
discriminator loss function)
9/17
![Page 10: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/10.jpg)
Results
Datasets MNIST-70k Reuters-10k HHAR
DAC EC (Ensemble Clustering) 96.50 73.34 81.24DAC 94.08 72.14 80.5
GMVAE 88.54 - -
DEC 84.30 72.17 79.86
AE + GMM (full covariances, median accuracy over 10 runs) 82.56 70.12 78.48
GMM 53.73 54.72 60.34
KM 53.47 54.04 59.98
Table: Experimental accuracy results (%, the higher, the better) based onthe Hungarian method
10/17
![Page 11: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/11.jpg)
Visualizations
11
3
2
23
7
3
2
5
6
6710
0
2
3
1
0
1
0
1
7486
0
59
69
90
20
22
22
84
6863
173
68
127
235
17
2
94
0
6819
42
8
22
318
13
80
7
7
6747
2
13
88
1
44
170
2
124
6104
4
125
3
2
43
3
17
0
6678
39
11
1
3
8
49
104
15
7025
0
1
17
24
29
25
2
62
6281
15
21
29
1
81
27
80
8
6230
20
59
0
10
18
3
4
1
0
11
3
2
23
7
3
2
5
6
6710
0
2
3
1
0
1
0
1
7486
0
59
69
90
20
22
22
84
6863
173
68
127
235
17
2
94
0
6819
42
8
22
318
13
80
7
7
6747
2
13
88
1
44
170
2
124
6104
4
125
3
2
43
3
17
0
6678
39
11
1
3
8
49
104
15
7025
0
1
17
24
29
25
2
62
6281
15
21
29
1
81
27
80
8
6230
20
59
0
10
18
3
4
1
0
9
8
7
6
5
4
3
2
1
0
0 1 2 3 4 5 6 7 8 9
Predicted class
Actu
al cla
ss
Figure: Confusion matrix for DAC on MNIST. (best seen in color)
11/17
![Page 12: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/12.jpg)
Visualizations
µk
µk + 0.5σ
µk + 1σ
µk + 1.5σ
µk + 2σ
µk + 2.5σ
µk + 3σ
µk + 3.5σ
Figure: Generated digits images. From left to right, we have the tenclasses found by DAC and ordered thanks to the Hungarian algorithm.From top to bottom, we go further and further in random directions fromthe centroids (the �rst row being the decoded centroids).
12/17
![Page 13: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/13.jpg)
Visualizations
Figure: Principal Component Analysis rendering of the code space forMNIST at the end of the DAC optimization, with colors indicating thetrue labels. (best seen in color)
13/17
![Page 14: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/14.jpg)
Conclusion
Representation Learning and Clustering operate a symbiosis
14/17
![Page 15: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/15.jpg)
References I
Christopher M.. Bishop.
Pattern recognition and machine learning.
Springer, 2006.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Deep Learning.
MIT Press, 2016.
http://www.deeplearningbook.org.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu,
David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua
Bengio.
Generative adversarial nets.
In Advances in Neural Information Processing Systems, pages
2672�2680, 2014.
15/17
![Page 16: Deep Adversarial Gaussian Mixture Auto-Encoder …harchaoui.org/warith/dac_ppt.pdfOptimization Scheme Input Space Code Space Gaussian Clusters ( ˇ, , ) Encoder Decoder GMM Discriminator](https://reader030.vdocuments.us/reader030/viewer/2022040403/5e87d1b68918d263b51cf202/html5/thumbnails/16.jpg)
References II
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian
Goodfellow.
Adversarial autoencoders.
arXiv preprint arXiv:1511.05644, 2015.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua
Bengio, and Pierre-Antoine Manzagol.
Stacked denoising autoencoders: Learning useful
representations in a deep network with a local denoising
criterion.
Journal of Machine Learning Research, 11(Dec):3371�3408,
2010.
16/17