Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Yiyi Liao*, Katja Schwarz*, Lars Mescheder, Andreas Geiger

Motivation3D Controllable Image Synthesis

3D controllability is essential in many applications, e.g., gaming, simulation, virtualreality and data augmentation3D controllable properties: 3D pose, shape, appearance of multiple objects andcamera viewpoint

MotivationClassical Rendering Pipeline

Render3D Design

3D ControllableExpensive and inefficient to design 3D models

Motivation2D Generative Models

2D Generator

Sample

Efficient, learned from only 2D imagesGeometry and appearance not disentangled → no 3D control

MotivationOurs

3D Generator

Render2D

GeneratorSample

3D ControllableEfficient, learned from only 2D imagesUnsupervised, disentangled 3D representation learning

MotivationOurs

3D Generator

Render2D

GeneratorSample

Idea: Learning the image synthesis pipeline jointly in 3D and 2D space

Method Overview 

3D Generator Differentiable Rendering 2D Generator

Render

Render

Render

Render

Alpha

Composition

Real

Background

Image

BackgroundImage

Real

ExperimentsDatasets

Car w/o BG Car with BG Indoor Fruit

ExperimentsCar Dataset

Layout2Im and our 2D baseline areonly controllable for 2D translationLayout2Im fails to disentangle objectidentity and poseOur method is controllable for 3Dtranslation and rotation withcoherent object identity

ExperimentsIndoor Dataset

Layout2Im and our 2D baseline areonly controllable for 2D translationLayout2Im fails to disentangle objectidentity and poseOur method is controllable for 3Dtranslation and rotation withcoherent object identity

ExperimentsFruit Dataset

We collect 800 images with 5 fruitsand 5 backgroundsOur method is able to synthesizeplausible images from real data

Failure Cases

A single primitive generates multiple objects occasionally

Identity might flip wrt. large viewpoint change

Stronger inductive biases are required to tackle these problems

Thank you!