generative adversarial networks - github pagesgenerative adversarial networks. abstracting a bit the...

Generative Adversarial NetworksPrem Seetharaman, CS349

1

Generative models

Generative models are unsupervised. Their only task is to generate samples that look like real samples.

Real samples of faces ->

Generative models

Generative models are unsupervised. Their only task is to generate samples that look like real samples.

Real samples of faces -> Generated faces!

Generative models

With generative models, we can leverage unlabeled training data.

Generative models

Some tasks have many solutions.

Example: generating speech for the same text in different styles.

Generative models

Example: Multiple translations for a given text.

Generative models

Central question: how do we model our training data?

Generative models

In these faces for example, there are several “latent” factors at play. What are they?

Distributions

Recall that training example - an image, an audio clip, etc. - can be thought of as a point in space.

Distributions

Let’s put all of our training data into this space.

Matching distributions

Then, generating data means to sample from this space such that the sampled points are near real points.

Red: sampled points Blue: real points


Recall that one way we did that was by fitting a GMM to the space and sampling from it.

GMM w/ 1 component


As the number of components gets higher, the sampled points start to overlap with the real points.

GMM w/ 3 components

15

GMM with 1 component

16

GMM with 10 components

17

GMM with 50 components

Lots of ways to make this happen

Many other methods for generative modeling: 1. Variational autoencoders 2. Autoregressive models 3. Generative adversarial

networks

Abstracting a bitThe goal is to be able to construct points similar to the real points.

Generator

Abstracting a bit

This is a good generator.

Generator

Abstracting a bit

This is a bad generator.

Generator

Abstracting a bitGenerators can produce multiple points that should hopefully be diverse.

Generator

Abstracting a bitA generator that can only produce one “type” of real sample is bad. Called mode collapse.

Generator

Abstracting a bitA generator that covers all of your training data is good! It produces diverse samples.

Generator

A loss function for generators?So we have some definition for what makes a generator “good” or “bad”. How do we turn this into a loss function?

Generator

Enter the discriminatorThe discriminator tries to tell a generated sample apart from a real sample.

Discriminator

Enter the discriminatorThe discriminator tries to tell whether a sample is real or fake.

Discriminator REAL


Discriminator FAKE


Discriminator REAL


Discriminator FAKE

The whole pictureGenerator attempts to create a data point.

DiscriminatorGenerator

The whole pictureIf this generated point is NOT close to the real points in the space…


The whole pictureThe discriminator will easily say that it’s a fake point.

DiscriminatorGenerator FAKE

The whole pictureBut if the generated point IS close to a real point…


The whole pictureThen the discriminator might be fooled!

DiscriminatorGenerator REAL

Generator’s goalFool the discriminator into thinking the generated points are real.


Discriminator’s goalDon’t be fooled by the generator.


Generator takes as input a latent codeThe output of the generator is conditioned on random noise.


Late

nt c

ode

Generator takes as input a latent codeThe generator interprets this random noise as a point on a manifold.


Late

nt c

ode

Generator takes as input a latent code…and maps points on the manifold to a point in the space of the data.


Late

nt c

ode

Key points

Generator

Latent code

To create good fake data, a generator must know what real data looks like.

The generator attempts to generate samples that are likely under the true distribution.

The distribution is learned implicitly via the discriminator.

Key points

The discriminator learns how to tell real data from fake data.

Thus, a GAN is like learning your loss function, rather than keeping it fixed! Discriminator

REAL or FAKE

This is a game with two players!Player 1: The Generator Player 2: The Discriminator


Late

nt c

ode

This is a game with two players!Player 1: The Generator Player 2: The Discriminator


Late

nt c

ode

Generator is a deep network!

Discriminator is a deep network!

This is a game with two players!Each player seeks to maximize its own success and minimize the success of the other -> a minimax game


Late

nt c

ode



This is a game with two players!The two players are at odds with one another and are trained in an alternating fashion.


Late

nt c

ode



Keep the generator fixed. Generate a bunch of fake points.


Late

nt c

ode

At first, the generator has no idea how to create fake points. The data is not close to the real points.

Take a bunch of real points…


Late

nt c

ode

…and feed them to the discriminator. The discriminator should output that these are real points. Use this as a training signal and backprop to train the discriminator.

FAKE

REAL

FAKE

REAL

Take a bunch of real points…


Late

nt c

ode

…and feed them to the discriminator. The discriminator should output that these are real points. Use this as a training signal and backprop to train the discriminator.

FAKE

REAL

FAKE

REAL

Backprop“REAL/FAKE” loss to discriminator

Take a bunch of fake points…


Late

nt c

ode

…and feed them to the discriminator. The discriminator should output that these are fake points.

REAL

FAKE

REAL

FAKE


Now keep discriminator fixed. Generate fake points again…


Late

nt c

ode

This time we will update the generator instead. Note how different points may be generated this time - this is due to the randomness of the latent code.

Take these fake points…


Late

nt c

ode

…and feed them to the discriminator. This time, we want the generator to fool the discriminator.

FAKE

FAKE

FAKE

FAKE

FAKE

Take these fake points…


Late

nt c

ode

However the generator doesn’t know the true data distribution yet! So the discriminator easily catches that we are making forgeries.

FAKE

FAKE

FAKE

FAKE

FAKE

Update the generator.


Late

nt c

ode

Update the generator by backpropping through the discriminator.

Backprop negation of “REAL/FAKE” loss to generator.

FAKE

FAKE

FAKE

FAKE

FAKE

Alternate between generator updates and discriminator updates.


Late

nt c

ode

REAL

REAL

FAKE

REAL

FAKE


Train the discriminator to tell fake points from real points…



Late

nt c

ode

Generate new fake points…



Late

nt c

ode

FAKE

FAKE

REAL

REAL

REAL

Backprop negation of “REAL/FAKE” loss to generator.

Update the generator to fool the discriminator.



Late

nt c

ode

As this process continues…



Late

nt c

ode

…the generator will start to put its generated points near the real points…



Late

nt c

ode

..leading to the discriminator to believe they are real points!

REAL

REAL

REAL

REAL

REAL

Time to add a bit of math

Credit to Bhiksha Raj at CMU for these images!



Intuition: how do you tell two coins apart?

Flip them and compare them to what you expect.



Green: you want high likelihood that x is real (because it is).

Red: you want low likelihood that G(z) is real (because it isn’t).



D(x) -> coin flip for D on x

D(G(z)) -> coin flip for D on G(x)



Expect 1, try to get 1

Expect 0, try to get 0

generative adversarial networks - github pagesgenerative adversarial networks. abstracting a bit the...

Documents