for 3 d c ont r ol l a bl e i m a ge s y nt h e s i s towa r ds u ns upe … · 2020-05-09 · towa...
Post on 11-Jul-2020
3 Views
Preview:
TRANSCRIPT
Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
Yiyi Liao*, Katja Schwarz*, Lars Mescheder, Andreas Geiger
Motivation3D Controllable Image Synthesis
3D controllability is essential in many applications, e.g., gaming, simulation, virtualreality and data augmentation3D controllable properties: 3D pose, shape, appearance of multiple objects andcamera viewpoint
MotivationClassical Rendering Pipeline
Render3D Design
3D ControllableExpensive and inefficient to design 3D models
Motivation2D Generative Models
2D Generator
Sample
Efficient, learned from only 2D imagesGeometry and appearance not disentangled → no 3D control
MotivationOurs
3D Generator
Render2D
GeneratorSample
3D ControllableEfficient, learned from only 2D imagesUnsupervised, disentangled 3D representation learning
MotivationOurs
3D Generator
Render2D
GeneratorSample
Idea: Learning the image synthesis pipeline jointly in 3D and 2D space
Method Overview
3D Generator Differentiable Rendering 2D Generator
Render
Render
Render
Render
Alpha
Composition
Real
Background
Image
BackgroundImage
Real
ExperimentsDatasets
Car w/o BG Car with BG Indoor Fruit
ExperimentsCar Dataset
Layout2Im and our 2D baseline areonly controllable for 2D translationLayout2Im fails to disentangle objectidentity and poseOur method is controllable for 3Dtranslation and rotation withcoherent object identity
ExperimentsIndoor Dataset
Layout2Im and our 2D baseline areonly controllable for 2D translationLayout2Im fails to disentangle objectidentity and poseOur method is controllable for 3Dtranslation and rotation withcoherent object identity
ExperimentsFruit Dataset
We collect 800 images with 5 fruitsand 5 backgroundsOur method is able to synthesizeplausible images from real data
Failure Cases
A single primitive generates multiple objects occasionally
Identity might flip wrt. large viewpoint change
Stronger inductive biases are required to tackle these problems
Thank you!
top related