generative - idc.ac.il · metz, luke, et al. "unrolled generative adversarial networks."...
TRANSCRIPT
![Page 1: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/1.jpg)
GenerativeAdversarialNetworks
Yoav Orlev - Deep Image Processing Seminar
![Page 2: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/2.jpg)
Generative ModelsAnd why we want them
![Page 3: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/3.jpg)
Discriminative vs Generative
![Page 4: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/4.jpg)
Discriminative vs Generative
![Page 5: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/5.jpg)
What are generative models?
Given a training set drawn from some distribution,
Tries to fit a model to best represent the data probability.
![Page 6: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/6.jpg)
What are generative models?
Given a training set drawn from some distribution,
Tries to fit a model to best represent the data probability.
![Page 7: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/7.jpg)
Why do we want generative models?
![Page 8: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/8.jpg)
Why do we want generative models?
“What I cannot create
I can not understand”*Richard Feynman
![Page 9: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/9.jpg)
Why do we want generative models?
● Understanding and Compressing Knowledge
![Page 10: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/10.jpg)
Why do we need generative models?
Radford et al (2015).
100 Million Parameters
● Understanding and Compressing Knowledge
![Page 11: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/11.jpg)
DCGAN Radford et al.
200GB -> 10MB
Generated Results From The ImageNet Dataset
![Page 12: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/12.jpg)
Why do we want generative models?
● Understanding and Compressing Knowledge
● Semi supervised Learning
![Page 13: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/13.jpg)
Why do we want generative models?
● Semi supervised Learning
Assumes some
Structure to the underlying
distribution of the data.
![Page 14: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/14.jpg)
Why do we want generative models?
● Semi supervised Learning
99.14% Accuracy on MNIST
Only 10 labels per class.
OpenAI Research
![Page 15: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/15.jpg)
Why do we need generative models?
● Understanding and Compressing Knowledge
● Semi supervised Learning
● Multi modal outputs
![Page 16: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/16.jpg)
Why do we need generative models?
● Multi modal outputs
Lotter et al. (2015)
![Page 17: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/17.jpg)
Why do we need generative models?
● Understanding and Compressing Knowledge
● Semi supervised Learning
● Multi modal outputs
● Generating Data!
![Page 18: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/18.jpg)
Why do we need generative models?
● Understanding and Compressing Knowledge
● Semi supervised Learning
● Multi modal outputs
● Generating Data!
● And more...
![Page 19: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/19.jpg)
Generative Model Types
● Naive Bayes
● Variational Autoencoders
● Hidden Markov Models
● Generative Adversarial Network
● And more...
![Page 20: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/20.jpg)
Generative Model Types
● Naive Bayes
● Variational Autoencoders
● Hidden Markov Models
● Generative Adversarial Network
● And more...
![Page 21: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/21.jpg)
Generative adversarial networksQuick Introduction
![Page 22: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/22.jpg)
End Goal
Training set
![Page 23: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/23.jpg)
End Goal
Input Output
1
2
7
5
2
1
code
![Page 24: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/24.jpg)
End Goal
Input Output
8
2
7
9
3
4
code
![Page 25: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/25.jpg)
How do we learn to generate?
![Page 26: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/26.jpg)
![Page 27: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/27.jpg)
Every Month Different Show.
![Page 28: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/28.jpg)
Every Month Different Show.
![Page 29: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/29.jpg)
![Page 30: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/30.jpg)
![Page 31: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/31.jpg)
![Page 32: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/32.jpg)
Concert name 1
![Page 33: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/33.jpg)
![Page 34: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/34.jpg)
You can’t come in!The ticket color should be red!
![Page 35: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/35.jpg)
Concert name 2
![Page 36: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/36.jpg)
You can’t come in!The ticket date should be in Arial font!
![Page 37: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/37.jpg)
Concert name 3
![Page 38: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/38.jpg)
You can’t come in!The ticket … should be ...!
Concert name x
![Page 39: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/39.jpg)
Concert name 1011
![Page 40: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/40.jpg)
Come In!
![Page 41: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/41.jpg)
Credits
DiscriminatorGenerator Training Data
![Page 42: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/42.jpg)
Notice!
Generator Training Data
X
![Page 43: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/43.jpg)
Generative adversarial networksIan J. Goodfellow et el. [2014]
![Page 44: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/44.jpg)
GAN has 2 players
D Discriminator
GeneratorG
![Page 45: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/45.jpg)
GAN has 2 players
D Discriminator
GeneratorG
Given a sample x, Outputs the probability of x coming from the training set
![Page 46: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/46.jpg)
GAN has 2 players
D Discriminator
GeneratorG
Given a sample x, Outputs the probability of x coming from the training set
Given a random z, Output an imageWhich look as if it came from the training
set
![Page 47: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/47.jpg)
GAN Architecture
Gz
G(z)
x
D
D(x)
D(G(z)
Slide taken from Kevin Mcguinness -
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
![Page 48: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/48.jpg)
GAN Architecture
Gz
G(z)
x
D
D(x)
D(G(z)
Slide taken from Kevin Mcguinness -
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
1
0
![Page 49: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/49.jpg)
GAN Training
Gz
G(z)
x
D
D(x)
D(G(z)
Slide taken from Kevin Mcguinness -
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
![Page 50: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/50.jpg)
GAN Training
Gz
G(z)
x
D
D(x)
D(G(z)
Slide taken from Kevin Mcguinness -
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
![Page 51: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/51.jpg)
GAN Process
![Page 52: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/52.jpg)
GAN Process
![Page 53: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/53.jpg)
GAN Process
![Page 54: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/54.jpg)
GAN Process
![Page 55: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/55.jpg)
GAN Process
![Page 56: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/56.jpg)
GAN Architecture
Slide taken from :
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
![Page 57: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/57.jpg)
Loss Function - MiniMax Game
![Page 58: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/58.jpg)
Loss Function - MiniMax Game
![Page 59: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/59.jpg)
Mini-Max Game
What is the max value V(D, G) can take?
![Page 60: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/60.jpg)
Mini-Max Game
What is the max value V(D, G) can take? 0
![Page 61: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/61.jpg)
Mini-Max Game
What is the max value V(D, G) can take? 0
What is the min value V(D, G) can take?
![Page 62: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/62.jpg)
Mini-Max Game
What is the max value V(D, G) can take? 0
What is the min value V(D, G) can take? -
8
![Page 63: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/63.jpg)
Why D wants to maximize
![Page 64: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/64.jpg)
Why D wants to maximize
training set
generated sample
![Page 65: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/65.jpg)
Why D wants to maximize
training set
generated sample
![Page 66: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/66.jpg)
Why G wants to Minimize
generated sample
![Page 67: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/67.jpg)
Why G wants to Minimize
generated sample
![Page 68: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/68.jpg)
Loss - In practice
Discriminator Generator
Given (x, 1) the loss is :
Given (x=G(z), 0) the loss is :
Given (G(z), 1) the loss is :
So basically we have cross entropy loss :
-
-
In practice we would maximize :
![Page 69: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/69.jpg)
Game Equilibrium
log(x)
0.5
log(1-x)
![Page 70: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/70.jpg)
GAN Recap
Gz
G(z)
x
D
D(x)
D(G(z)
Slide taken from Kevin Mcguinness -
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016
![Page 71: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/71.jpg)
GAN Algorithm
![Page 72: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/72.jpg)
Process
![Page 73: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/73.jpg)
Generative Adversarial NetworksResults
![Page 74: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/74.jpg)
Original GAN (2014)
Goodfellow et el. (2014)
![Page 75: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/75.jpg)
DCGAN (2015)
Radford et el. (2014)
![Page 76: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/76.jpg)
PPGN (2016)
Nguyen et el. (2016)
![Page 77: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/77.jpg)
Progressive Growing of GANs (2018)
Karras et el. (2018)
![Page 78: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/78.jpg)
Generative adversarial networksProblems
![Page 79: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/79.jpg)
Non Convergence
Unlike traditional Deep Learning approaches GAN involve two players
● D is trying to maximize its reward● G is trying to minimize D’s reward
● SGD was not designed to find the NE of a game● Might not converge!
![Page 80: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/80.jpg)
Non Convergence example
![Page 81: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/81.jpg)
Non Convergence example
● State 1: [x > 0 | y > 0 | V > 0]
Increase y | Decrease x
![Page 82: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/82.jpg)
Non Convergence example
● State 1: [x > 0 | y > 0 | V > 0]
Increase y | Decrease x
● State 2: [x < 0 | y > 0 | V < 0]
Decrease y | Decrease x
![Page 83: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/83.jpg)
Non Convergence example
● State 1: [x > 0 | y > 0 | V > 0]
Increase y | Decrease x
● State 2: [x < 0 | y > 0 | V < 0]
Decrease y | Decrease x
● State 3: [x < 0 | y < 0 | V > 0]
![Page 84: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/84.jpg)
Non Convergence example
● State 1: [x > 0 | y > 0 | V > 0]
Increase y | Decrease x
● State 2: [x < 0 | y > 0 | V < 0]
Decrease y | Decrease x
● State 3: [x < 0 | y < 0 | V > 0]
![Page 85: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/85.jpg)
Non Convergence example
● State 1: [x > 0 | y > 0 | V > 0]
Increase y | Decrease x
● State 2: [x < 0 | y > 0 | V < 0]
Decrease y | Decrease x
● State 3: [x < 0 | y < 0 | V > 0]
![Page 86: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/86.jpg)
Mode collapse
![Page 87: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/87.jpg)
Mode collapse
![Page 88: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/88.jpg)
Mode collapse
![Page 89: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/89.jpg)
Progressive growing of GANs for improved quality, Stability and variation.Karras, Tero, et al. (2018)
![Page 90: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/90.jpg)
Progressive Growing
![Page 91: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/91.jpg)
Smooth Transition
![Page 92: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/92.jpg)
Network architecture and learning
![Page 93: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/93.jpg)
Network architecture and learning
![Page 94: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/94.jpg)
Network architecture and learning
![Page 95: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/95.jpg)
Network architecture and learning
![Page 96: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/96.jpg)
Network architecture and more
Minibatch Standard Deviation
![Page 97: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/97.jpg)
Results
![Page 98: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/98.jpg)
Results
![Page 99: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/99.jpg)
Results
![Page 100: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/100.jpg)
Thank You!
![Page 101: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/101.jpg)
References
![Page 102: Generative - idc.ac.il · Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016). Zhu, Jun-Yan, et al. "Unpaired image-to-image translation](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec678bb12b5ea15691974de/html5/thumbnails/102.jpg)
● Metz, Luke, et al. "Unrolled Generative Adversarial Networks." arXiv preprint arXiv:1611.02163 (2016).
● Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." arXiv preprint
arXiv:1703.10593 (2017).
● Karras, Tero, et al. "Progressive growing of gans for improved quality, stability, and variation." arXiv preprint arXiv:1710.10196
(2017).
● Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014.