addressing gan limitations: resolution, lack of novelty ... › semester2019 › ... · resolution,...
TRANSCRIPT
Addressing GAN limitations: resolution, lack of novelty and control on generations
Camille Couprie
Facebook AI Research
Joint works with O. Sbai, M. Aubry, A, Bordes, M. Elhoseiny, M. Riviere, Y.
LeCun, M. Mathieu, P. Luc, N. Neverova, J. Verbeek.
2019
1
Why do we care about generative models
2
• Scene Understanding can be assessed by checking the ability to
generate plausible new scenes.
• Generative models are interesting if they can be used to go beyond
training data: data of higher resolution, data augmentation to help
train better classifiers, use the learned representations in other
tasks, or make prediction about uncertain events...
• 1/ Design inspiration from adversarial generative networks
• 2/ High resolution, decoupled generation
• 3/ Vector image generation by learning parametric layer
decomposition
• 4/ Future frame prediction
Outline
3
4
Sbai, Elhoseiny, Bordes, LeCun, Couprie, ECCV workshop 17
Design inspiration from generative networks
N o v e l t y
H e d o n i c
Va l u e
1 2 30 4
RANDOM NUMBERS
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORKGenerator
FakeGENERATED
IMAGE
Generative Adversarial Networks
Goodfellow et al, 2014
RANDOM NUMBERS
Real INPUT
Generator
Real
AdVERSARIAL
NETWORK
GENERATED
IMAGE
0 . 3
0 . 7
0 . 1
0 . 8
Generative Adversarial Networks
Goodfellow et al, 2014
Deep convolutional GANsRADFORD ET AL : ICLR 2015
Training with pictures of about 2000 Clothing items
Floral StripedTiled Uniform Dotted Animal Print Graphical
Texture and shape labels
DressSkirt JacketPullover T-Shirt Coat Top
RANDOM NUMBERS
Real INPUT
GeneratorAdVERSARIAL
NETWORK
GENERATED
IMAGE
0 . 3
0 . 7
0 . 1
0 . 8
S h a p e C L A S S
0 / 1
( R E A L / F A K E )
T E X T U R E C L A S S
Class conditioned GAN
• Generator’s
loss
• Discriminator’s
loss
• Auxiliary
classifier
discriminator:
• Additional loss
for the generator:
GAN Optimization objectives
Without conditioning With class conditioning
RANDOM
NUMBERSGenerator
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORK
Generated
Image
Dotted
Floral
graphical
uniform
tiled
striped
Animal print
1 0 0 %
Introduction of a Style Deviation criterion
8 %
6 %
7 %
5 7 %
6 %
1 1 %
9 %
RANDOM
NUMBERSGenerator
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORK
Generated
Image
Dotted
Floral
graphical
uniform
tiled
striped
Animal print
Introduction of a Style Deviation criterion
RANDOM
NUMBERSGenerator
0 . 3
0 . 7
0 . 1
0 . 8
AdVERSARIAL
NETWORKGenerated Image
D o t t e d
F l o r a l
g r a p h i c a l
u n i f o r m
t i l e d
s t r i p e d
A n i m a l p r i n t
With the Style
Deviation criterion
(CAN H)
Multi-class cross entropy loss:
Binary cross entropy loss :
Tested deviation objectives
O v e r a l L L i k a b i l i t y ( % )
6 5 7 0 7 56 08
0
CAN: GAN with Creativity loss, (H) stands for the use of a holistic loss.
CAN
texTUREGAN
Style
CAN(h)
texTURE
CAN(h)
texTUREr e a l i s t i c
A p p e a r a n c e
4 8
5 3 . 5
5 9
6 4 . 5
7
0
Human Evaluation Study
Models with texture deviation are Most Popular
N o v e l t y0 1
L i k e a b i l i t y
6 2
6 6
7 0
7 4
7
8
judged by humans and measured as a distance to similar training images
Can
texture
Can (H)
Shape Can (H)
texture
style
can (h)
GAN
2
0
M. Riviere, C. Couprie, Y. LeCun
Decoupled adversarial image generation
Motivation:
- Take advantage of white background clothing datasets
- Potentially avoid defaults in generated shapes
- Better enforce shape conditioning of generations
1024x1024 generations on the RTW dataset
2
1
Using Morgane’s pytorch “progressive growing of GANs” available online,
Karras et al., ICLR’18
Decoupled architecture
2
2
Shape
Generator
Texture
Generator
Generation
Real image
Shape and
pose classes
Texture, color, shape and pose classes
Discriminator
Real / Fake
Real image
Discriminator
Real / Fake
Texture, color,
shape and
pose classes
×
DgDs
Gs
Gt
z
Random generations
2
3
Progressive growingProgressive growing with decoupled
architecture
Better class conditioning
2
4
Accuracy of classifiers trained on FashionGen
Clothing and FashionGen Shoes on our different
models results (GAN-test metric)
=
+12%
+14%
-12%
Overall average improvement: 4.7%
2
5
Sbai, Couprie, Aubry, arxiv dec18
Vector Image Generation
by learning parametric layer decomposition
Current deep generative models
are great but…
… are limited in resolution, and
control in generations
Kanan et al: Layered
GANs (LR-GANs), ICLR’17
GANIN et al. SPIRAL,
ICML’18
Related work
2
6
Our approach
2
7
Spoiler alert: yes, we can
generate sharper images,
this is just an example.
Iterative generation : !" = $(!, !"'()
Vectorized mask generation *"(+, ,) = $(+, ,, -")
Alpha-blending !"(+, ,) = !"'( +, , . (1-*"(+, ,))+ /"*"(+, ,)
Our iterative pipeline for image reconstruction
2
8
x x
×<latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit>
×<latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit><latexit sha1_base64="L3dZ0XgOyNX2PKpIro/rl+dxAfQ=">AAACBXicbVDLSgMxFL3js46vqks3wSK4KjMi6LLgxpVUsA9oS8lkMm1skhmTjDAMXbt1q//gTtz6Hf6CX2HazsK2HggczrmXc3OChDNtPO/bWVldW9/YLG252zu7e/vlg8OmjlNFaIPEPFbtAGvKmaQNwwyn7URRLAJOW8HoeuK3nqjSLJb3JktoT+CBZBEj2Fip2TVMUN0vV7yqNwVaJn5BKlCg3i//dMOYpIJKQzjWuhNx+ih7bo6VYYTTsdtNNU0wGeEB7VgqsY3p5dN7x+jUKiGKYmWfNGiq/t3IsdA6E4GdFNgM9aI3Ef/zOqmJrno5k0lqqCSzoCjlyMRo8nkUMkWJ4ZklmChmb0VkiBUmxlY0l6KNwCpT4di15fiLVSyT5nnV96r+3UWldlvUVIJjOIEz8OESanADdWgAgQd4gVd4c56dd+fD+ZyNrjjFzhHMwfn6BRxkmZg=</latexit>
+<latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit><latexit sha1_base64="CuHZZdZlV0qZYt64uTfAQB24j90=">AAACAHicbVDLSsNAFJ3UV42vqks3g0UQhJKIoMuCG1fSgn1AG8pkctMOnUzizEQIoRu3bvUf3Ilb/8Rf8CuctlnY1gMXDufcy733+AlnSjvOt1VaW9/Y3Cpv2zu7e/sHlcOjtopTSaFFYx7Lrk8UcCagpZnm0E0kkMjn0PHHt1O/8wRSsVg86CwBLyJDwUJGiTZS82JQqTo1Zwa8StyCVFGBxqDy0w9imkYgNOVEqV7I4VF4dk6kZpTDxO6nChJCx2QIPUMFiUB5+ezUCT4zSoDDWJoSGs/UvxM5iZTKIt90RkSP1LI3Ff/zeqkOb7yciSTVIOh8UZhyrGM8/RsHTALVPDOEUMnMrZiOiCRUm3QWtigdEZnJYGKbcNzlKFZJ+7LmOjW3eVWt3xcxldEJOkXnyEXXqI7uUAO1EEWAXtArerOerXfrw/qct5asYuYYLcD6+gWfb5cT</latexit>
ct<latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit><latexit sha1_base64="bbRbyyomJJsasHk6yTZKV4PERh8=">AAACC3icbVDLSsNAFJ3UV42vqks3g0VwVRIRdFlw40oq2Ac0oUwmk3bozCTOTIQQ8glu3eo/uBO3foS/4Fc4abOwrQcuHM65l3s4QcKo0o7zbdXW1jc2t+rb9s7u3v5B4/Cop+JUYtLFMYvlIECKMCpIV1PNyCCRBPGAkX4wvSn9/hORisbiQWcJ8TkaCxpRjLSRPI8jPQmiHBcjPWo0nZYzA1wlbkWaoEJn1PjxwhinnAiNGVJqGDHyKHw7R1JTzEhhe6kiCcJTNCZDQwXiRPn5LHUBz4wSwiiWZoSGM/XvRY64UhkPzGaZUi17pfifN0x1dO3nVCSpJgLPH0UpgzqGZQUwpJJgzTJDEJbUZIV4giTC2hS18EVpjmQmw8I25bjLVayS3kXLdVru/WWzfVfVVAcn4BScAxdcgTa4BR3QBRgk4AW8gjfr2Xq3PqzP+WrNqm6OwQKsr18ScpxS</latexit>
ctMt<latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit><latexit sha1_base64="TJwISkHzUulgDYQt6eHG/gsMlR0=">AAACEXicbVDLSsNAFJ3UV62vqEs3g0VwVRIRdFlw40apYB/QhjCZTNqhk0mcuSmU0K9w61b/wZ249Qv8Bb/CSduFbT1w4XDOvdzDCVLBNTjOt1VaW9/Y3CpvV3Z29/YP7MOjlk4yRVmTJiJRnYBoJrhkTeAgWCdVjMSBYO1geFP47RFTmifyEcYp82LSlzzilICRfNvuxQQGQZTTiQ/4zgffrjo1Zwq8Stw5qaI5Gr790wsTmsVMAhVE624k2JP0KjlRwKlgk0ov0ywldEj6rGuoJDHTXj7NPsFnRglxlCgzEvBU/XuRk1jrcRyYzSKpXvYK8T+vm0F07eVcphkwSWePokxgSHBRBA65YhTE2BBCFTdZMR0QRSiYuha+aIiJGqtwUjHluMtVrJLWRc11au7DZbV+P6+pjE7QKTpHLrpCdXSLGqiJKBqhF/SK3qxn6936sD5nqyVrfnOMFmB9/QI5+J3r</latexit>
2
9
Our iterative pipeline for image generation
3
0
Training criteria
Adversarial net criterion: Wasserstein loss with Gradient Penalty (WGAN-GP),
Gulrajani et al. NIPS’17
Our generator loss in the GAN setting:
Our generator loss in the image reconstruction setting:
Results using a l1 reconstruction loss
3
1
Results using a l1 reconstruction loss
3
2
Target Our iterative reconstruction
Editing the original image using
extracted masks, by performing
local modifications of
luminosity (top), or color
modification using a blending
of masks (bottom).
Applications
3
3
Image editing using masks:
Using chosen extracted mask(s)
from image reconstruction, we
apply object removal (top)
and color modifications with
object removal (bottom).
Applications
3
4
Image vectorization:
Reconstruction results on
MNIST images. Our model
learns a vectorized mask
representation of digits that
can be generated at any
resolution without
interpolation artifacts.
Applications
3
5
RGB image
Baselines
3
6
RGB imagez
1/ MLP-baseline
MLPResnet
RGB image
Baselines
3
8
RGB imagez
1/ MLP-baseline
2/ ResNet baseline
ResnetResnet
RGB image
Baselines
3
9
RGB imagez
1/ MLP-baseline
2/ ResNet baseline
3/ MLP-xy baseline
MLP
(x,y)
Resnet
Comparative results
4
0
- Using a l1 reconstruction loss
- Parameters for our approach:
20 masks, p of size 10, c of size 3:
260 parameters
- Baselines: size of the latent
code z = 20x13=260
CelebA generations trained on 64x64
images, sampled at 256x256
GAN results
4
1
CIFAR10 generations trained on
32x32 images, sampled at 256x256
GAN
Results on
ImageNet
4
2
trained on
64x64
images,
sampled at
1024x1024
b
Results
4
4
Result
with
perceptual
loss
Perceptual
lossTarget
Conclusion and Future work
4
5
Faster training
Use class conditioning
Texture image generation
Predicting next frames in videos
Michael Mathieu, Camille Couprie, Yann LeCun, ICLR16
4 input images Our 2 predictions
Deep multiscale video prediction beyond Mean square error
• Result with a simple convolutional network trained minimizing an l2 loss
• Our result using
• A multiscale architecture
• an image gradient different loss
• Use adversarial training
4
8
P. Luc, N. Neverova, C. Couprie, J. Verbeek, Y. LeCun ECCV18
Predicting deeper into the future of semantic
segmentations
4 input images Our 2 predictions
• Predictions in the RGB space quickly become blurry despites previous attempts
• Idea: predict in the space of semantic segmentation
Approach – predicting deeper into the future
Single time-step
Autoregressive model
Batch model
St−3 St−2 St−1 St St+1
Lt+1
St−3 St−2 St−1 St St+1 St+2 St+3
Lt+1 Lt+2 Lt+3
St−3 St−2 St−1 St St+1 St+2 St+3
Lt+1 Lt+2 Lt+3
Autoregressive model is either :
• used for inference without
additional training (w.r.t. to single
time step model) AR
• Fine-tuned using BPTT
AR fine-tune
Autoregressive mode is only possible
for X2X, S2S, XS2XS
Same color = shared weights
4
9
Some results
50
Baselines :• Copy the last input frame to the
output
• Estimate flow between the two last inputs, and project the last input forward using the flow
Performance measure (mean IoU) of our approach and baselines on the Cityscapes dataset
Flow baseline
Our AutoRegressive fine-tune result
Instance level segmentation: Mask RCNN
• Extends Faster RCNN [Ren et al.’15] by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition
K. He G. Gkioxari P. Dollar R. Girshick’17
1
0
5
2
P. Luc, C. Couprie, Y. LeCun, J. Verbeek, ECCV18
Predicting future segmentation instances by
forecasting convolutional features
car 1.00
bicycle 0.92bicycle 0.97
person 0.99person 0.97person 0.98 person 0.93person 0.72
person 0.97
bicycle 1.00
person 0.76person 0.97
person 0.99person 0.98person 0.99person 0.94
person 0.91 person 1.00 person 0.56person 0.84
Luc, Neverova et al. ICCV17 F2F predictions
Conclusions
5
3
Some open problems:
- Automatic metrics to evaluate generative models performances
- Non deterministic training losses for future prediction
Torch code online :
- For future video prediction:
- Vector image generation: available soon on Othman Sbai’s github
- of RGBs : on Michael Mathieu’s github
- of semantic segmentations : on Pauline’s Luc github
- of instance segmentation : on Pauline’s Luc github