generative adversarial networksraul/rs/gans.pdf · generative adversarial network gans adopt the...
TRANSCRIPT
![Page 1: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/1.jpg)
GENERATIVE ADVERSARIAL NETWORKS
R. Q. FEITOSA
![Page 2: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/2.jpg)
The GAN Boom
2
source: GAN Zoo - https://github.com/hindupuravinash/the-gan-zoo
![Page 3: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/3.jpg)
Application Examples Image to Image translation
3
images from Isola, et al., 2017
![Page 4: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/4.jpg)
Application Examples Image captioning
4
from /
![Page 5: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/5.jpg)
Application Examples Edges to Image translation
5
Demo: https://affinelayer.com/pixsrv/
![Page 6: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/6.jpg)
Application Examples Edges to Image translation
6
Demo: https://affinelayer.com/pixsrv/
![Page 7: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/7.jpg)
Application Examples Edges to Image translation
7
Demo: https://affinelayer.com/pixsrv/
![Page 8: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/8.jpg)
Application Examples Edges to Image translation
8
Demo: https://affinelayer.com/pixsrv/
![Page 9: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/9.jpg)
𝑥~𝑝𝑑𝑎𝑡𝑎
𝑝𝑚𝑜𝑑𝑒𝑙
𝑦~𝑝𝑚𝑜𝑑𝑒𝑙
Generative Models
9
training samples generated samples
images credited by Goodfellow. 2016
learn the distribution
underlying the
training samples
generate new
samples from the
learned distribution
![Page 10: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/10.jpg)
Generative Models
10
training samples generated samples
learn the distribution
underlying the
training samples
generate new
samples from the
learned distribution
𝑥~𝑝𝑑𝑎𝑡𝑎
𝑝𝑚𝑜𝑑𝑒𝑙
𝑦~𝑝𝑚𝑜𝑑𝑒𝑙
if 𝑝𝑚𝑜𝑑𝑒𝑙 is a high
dimensional complex
distribution, there is no
direct way to do this
![Page 11: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/11.jpg)
Generative Adversarial Network
11
training samples generated samples
learn the distribution
underlying the
training samples
generate new
samples from the
learned distribution
𝑥~𝑝𝑑𝑎𝑡𝑎
𝑝𝑚𝑜𝑑𝑒𝑙
𝑦~𝑝𝑚𝑜𝑑𝑒𝑙 Transformation
![Page 12: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/12.jpg)
Generative Adversarial Network GANs adopt the following alternative approach:
Sample from a simple distribution, e.g., random noise, and learn a transformation to the target distribution
12
Question: How can we design such a complex
function that maps samples from a simple
distribution to a complex target distribution?
Answer: a neural network
z Generator
Network
input:
random noise
output:
sample from training distribution
Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014
![Page 13: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/13.jpg)
Training GANs: two-player game Generator network: try to fool the discriminator by generation realistic images
Discriminator network: try to distinguish between real and fake images
13
z
Discriminator
Network
Generator
Network
real image samples
generated image samples
real/fake random noise
![Page 14: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/14.jpg)
Training GANs: two-player game Generator network: try to fool the discriminator by generation realistic images
Discriminator network: try to distinguish between real and fake images
Train jointly in minimax game
Minmax objective function:
14
min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
Discriminator output
for real data 𝒙 Discriminator output for
fake data 𝑮 𝒛
Discriminator outputs likelihood in (0,1) real interval
Discriminator parameters Generator parameters
![Page 15: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/15.jpg)
Training GANs: two-player game Generator network: try to fool the discriminator by generation realistic images
Discriminator network: try to distinguish between real and fake images
Train jointly in minimax game
Minmax objective function:
• Discriminator (𝛉𝑑) wants to maximize objective such 𝐷𝛉𝑑 𝒙 is close to 1 for
real and 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛 is close to 0 for fake.
• Generator (𝛉𝑔) wants to minimize objective such that 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛 is
close to 1 (discriminator is fooled into thinking generated 𝐺𝛉𝑔 𝒛 is real)
15
min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
Discriminator output
for real data 𝒙 Discriminator output for
fake data 𝑮 𝒙
Discriminator outputs likelihood in (0,1) real interval
![Page 16: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/16.jpg)
Training GANs: two-player game Minmax objective function:
Alternate between:
1. Gradient ascent on discriminator (keep 𝛉𝑔 constant)
2. Gradient descent on generator (keep 𝛉𝑑 constant)
16
min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
min𝛉𝑔𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
At the beginning fake samples will be easily recognized by the
discriminator, but the gradient in this region is flat slow learning! 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
![Page 17: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/17.jpg)
Minmax objective function:
Alternate between:
1. Gradient ascent on discriminator (keep 𝛉𝑔 constant)
2. Instead: Gradient ascent on generator – change the objective
Training GANs: two-player game
17
min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎 𝒙 log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎 𝒙 log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
max𝛉𝑔𝔼𝒛~𝑝 𝒛 log 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛 Instead of minimizing likelihood of discriminant being correct,
maximize likelihood of discriminator being wrong fast learning!
. 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
log 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
![Page 18: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/18.jpg)
Training GANs: summary for number of training iterations do
for k steps do
• Sample minibatch of m noise samples {𝒛(1)…𝒛(𝑚)} from noise prior 𝑝𝑔 𝒛
• Sample minibatch of m examples {𝒙(1)…𝒙(𝑚)} from data generating
distribution 𝑝𝑑𝑎𝑡𝑎 𝒙
• Update the discriminator
end for
• Sample minibatch of m noise samples {𝒛(1)…𝒛(𝑚)} from noise prior 𝑝𝑔 𝒛
• Update the discriminator by ascending its stochastic gradient (improved
objective)
end for
18
𝛻𝜃𝑑1
𝑚 log𝐷𝜃𝑑 𝒙
𝑖 + log 1 − 𝐷𝜃𝑑 𝐺𝜃𝑔 𝒛𝑖
𝑚
𝑖=1
𝛻𝜃𝑔1
𝑚 log 𝐷𝜃𝑑 𝐺𝜃𝑔 𝒛
𝑖
𝑚
𝑖=1
![Page 19: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/19.jpg)
Generator Implementation DCGANS:
most “deconvs” are batch normalized.
19
Radfort et al. 2015, UNSUPERVISED REPRESENTATION LEARNING WITH DEEP CONVOLUTIONAL
GENERATIVE ADVERSARIAL NETWORKS
![Page 20: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/20.jpg)
Once trained… Generator network: try to fool the discriminator by generation realistic images
Discriminator network: try to distinguish between real and fake images
20
z
Discriminator
Network
Generator
Network
real image samples
generated image samples
real/fake random noise
After training, use
generator network to
generate new images.
![Page 21: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/21.jpg)
Application examples Source to target domain transfer
21
Zhu et al., 2018 , Unpaired Image-to-
Image Translation using Cycle-Consistent
Adversarial Networks
![Page 22: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/22.jpg)
Application examples Text to image synthesis
22
Reed et al., 2015, Generative Adversarial
Text to Image Synthesis
![Page 23: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/23.jpg)
Conditional GANs
23
Discriminator
Network
Generator
Network
real image sample
generated image sample
real/ fake
random noise
condition
condition
condition
z
![Page 24: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/24.jpg)
Conditional GANs
24
z
![Page 25: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/25.jpg)
Conditional GANs GAM
Conditional GAN
25
min𝛉𝑔max𝛉𝑑 𝔼𝒙~𝑝𝑑𝑎𝑡𝑎 𝒙 log 𝐷𝛉𝑑 𝒙
+ 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛
min𝛉𝑔max𝛉𝑑 𝔼𝒙,𝒚~𝑝𝑑𝑎𝑡𝑎 𝒙,𝒚 log 𝐷𝛉𝑑 𝒙, 𝒚
+ 𝔼𝒙~𝑝 𝒙 ,𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝒙, 𝐺𝛉𝑔 𝒙, 𝒛
condition real image
generated image
![Page 26: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/26.jpg)
Application examples Image-to-image translation
26
Zhu et al., 2018 , Towards Multimodal Image-to-Image Translation
![Page 27: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/27.jpg)
The GAN Zoo Lots of references on GANs
Click here to access The Gan Zoo
27
![Page 28: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/28.jpg)
GENERATIVE ADVERSARIAL NETWORKS
Thank you!
R. Q. FEITOSA
28
![Page 29: GENERATIVE ADVERSARIAL NETWORKSraul/RS/GANs.pdf · Generative Adversarial Network GANs adopt the following alternative approach: Sample from a simple distribution, e.g., random noise,](https://reader033.vdocuments.us/reader033/viewer/2022042303/5ecde342c9dc5a794236dd8e/html5/thumbnails/29.jpg)
References • Isola, P., Zhu, J. Y., Zhou, T., Efros, A. A., Image-to-Image Translation with
Conditional Adversarial Networks, CVPR, 2017, arXiv:1611.07004v2
29