![Page 1: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/1.jpg)
Adversarial Autoencoders
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly,Ian Goodfellow, Brendan Frey
Presented by: Paul Vicol
![Page 2: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/2.jpg)
Outline
Every single part of the movie was absolutely great!
● Adversarial Autoencoders○ AAE with continuous prior distributions○ AAE with discrete prior distributions○ AAE vs VAE
● Wasserstein Autoencoders○ Generalization of Adversarial Autoencoders○ Theoretical Justification for AAEs
![Page 3: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/3.jpg)
Regularizing Autoencoders
● Classical unregularized autoencoders minimize a reconstruction loss● This yields an unstructured latent space
○ Examples from the data distribution are mapped to codes scattered in the space○ No constraint that similar inputs are mapped to nearby points in the latent space○ We cannot sample codes to generate novel examples
● VAEs are one approach to regularizing the latent distribution
![Page 4: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/4.jpg)
Adversarial Autoencoders - Motivation
Every single part of the movie was absolutely great!● Goal: An approach to impose structure on the latent space of an autoencoder
● Idea: Train an autoencoder with an adversarial loss to match the distribution of the latent space to an arbitrary prior○ Can use any prior that we can sample from either continuous (Gaussian) or
discrete (Categorical)
![Page 5: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/5.jpg)
AAE Architecture
● Adversarial autoencoders are generative autoencoders that use adversarial training to impose an arbitrary prior on the latent code
Encoder / GAN Generator Decoder
Discriminator
-
+
![Page 6: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/6.jpg)
Training an AAE - Phase 1
1. The reconstruction phase: Update the encoder and decoder to minimize reconstruction error
Encoder / GAN Generator Decoder
![Page 7: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/7.jpg)
Training an AAE - Phase 2
2. Regularization phase: Update discriminator to distinguish true prior samples from generated samples; update generator to fool the discriminator
Encoder / GAN Generator
Discriminator
-
+
![Page 8: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/8.jpg)
AAE vs VAE
● VAEs use a KL divergence term to impose a prior on the latent space● AAEs use adversarial training to match the latent distribution with the prior
● Why would we use an AAE instead of a VAE?○ To backprop through the KL divergence we must have access to the functional
form of the prior distribution p(z)○ In an AAE, we just need to be able to sample from the prior to induce the latent
distribution to match the prior
Reconstruction Error KL Regularizer
Replaced by adversarial loss in AAE
![Page 9: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/9.jpg)
AAE vs VAE: Latent Space
● Imposing a Spherical 2D Gaussian prior on the latent space
AAE VAE
Gaps in the latent space; not well-packed
![Page 10: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/10.jpg)
AAE vs VAE: Latent Space
● Imposing a mixture of 10 2D Gaussians prior on the latent space
AAE VAE
VAE emphasizes the modes of the distribution; has systematic differences from the prior
![Page 11: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/11.jpg)
GAN for Discrete Latent Structure
● Core idea: Use a discriminator to check that a latent variable is discrete
![Page 12: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/12.jpg)
GAN for Discrete Latent Structure
● induces the softmax output to be highly peaked at one value● Similar to continuous relaxation with temperature annealing, but does not
require setting a temperature or annealing schedule
Without GAN Regularization With GAN Regularization
![Page 13: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/13.jpg)
Semi-Supervised Adversarial Autoencoders
● Model for semi-supervised learning that exploits the generative description of the unlabeled data to improve classification performance
● Assume the data is generated as follows:
● Now the encoder predicts both the discrete class y (content) and the continuous code z (style)
● The decoder conditions on both the class label and style vector
![Page 14: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/14.jpg)
Semi-Supervised Adversarial Autoencoders
![Page 15: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/15.jpg)
Semi-Supervised Adversarial AutoencodersImposes a discrete (categorical)
distribution on the latent class variable
Imposes a continuous (Gaussian) distribution on the latent style variable
![Page 16: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/16.jpg)
Semi-Supervised Classification Results
● AAEs outperform VAEs
![Page 17: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/17.jpg)
Unsupervised Clustering with AAEs
● An AAE can disentangle discrete class variables from continuous latent style variables without supervision
● The inference network predicts one-hot vector with K = num clusters
![Page 18: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/18.jpg)
Adversarial Autoencoder Summary
● Flexible approach to impose arbitrary distributions over the latent space● Works with any distribution you can sample from, continuous and discrete● Does not require temperature/annealing hyperparameters
● May be challenging to train due to the GAN objective● Not scalable to many latent variables → need a discriminator for each
Pros
Cons
![Page 19: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/19.jpg)
Wasserstein Auto-Encoders (Oral, ICLR 2018)
● Generative models (VAEs & GANs) try to minimize discrepancy measures between the data distribution and the model distribution
● WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution:
Regularizer encourages the encoded distribution to match the prior
Reconstruction cost
![Page 20: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/20.jpg)
WAE - Justification for AAEs
● Theoretical justification for AAEs:● When WAE = AAE● AAEs minimize the 2-Wasserstein distance between and
● WAE generalizes AAE in two ways:1. Can use any cost function in the input space2. Can use any discrepancy measure in the latent space
● Not just an adversarial one
![Page 21: Adversarial Autoencoders - GitHub Pages · Adversarial Autoencoders - Motivation Every single part of the movie was absolutely great! Goal: An approach to impose structure on the](https://reader030.vdocuments.us/reader030/viewer/2022041015/5ec5f59990ca1d693c706156/html5/thumbnails/21.jpg)
Thank you!