neurosymbolic 3d models...learning to generalize kinematic models to novel objects, abbatematteo et...
TRANSCRIPT
![Page 1: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/1.jpg)
Neurosymbolic 3D Models:
Learning to Generate 3D Shape Programs
Daniel Ritchie
![Page 2: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/2.jpg)
WHO AM I?
This guy!
![Page 3: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/3.jpg)
Brown University
• Located in Providence, Rhode Island• #14 University in the US (US News)
![Page 4: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/4.jpg)
Brown Computer Science Department
• 37 full-time faculty• 2-year Masters program• Fully-funded PhD program (5 years)• #25 for CS Graduate Study (US News)
![Page 5: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/5.jpg)
Brown Visual Computing
• Nine (9) faculty
• Active research in graphics,
vision, HCI, visualization, ...
• Regularly publish in top visual
computing venues
(SIGGRAPH, CVPR, ICCV, ...)
http://visual.cs.brown.edu/
![Page 6: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/6.jpg)
Brown Visual Computing
• Andy van Dam:
co-founder of ACM SICGRAPH
(pre-cursor to SIGGRAPH)
http://visual.cs.brown.edu/
![Page 7: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/7.jpg)
Brown Visual Computing
• Andy van Dam &
Spike Hughes:
Authors of
“Computer Graphics:
Principles and Practice”
http://visual.cs.brown.edu/
![Page 8: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/8.jpg)
My Research (Broadly)
Computer Graphics AI + ML
![Page 9: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/9.jpg)
My Research (Specifically)
Generate
Infer
3D Structures
• Objects• Scenes• ...
Generative Models
• Programs• Deep Networks• ...
What areneurosymbolic 3D models, and how do they relate to all of
this?
![Page 10: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/10.jpg)
FIRST, A LITTLE BACKGROUND& MOTIVATION...
![Page 11: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/11.jpg)
Increasing Demand for 3D Content
Traditional driver: Entertainment (Games, VR, ...)
![Page 12: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/12.jpg)
Increasing Demand for 3D Content
12
E-Commerce (esp. furniture / interior design)
![Page 13: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/13.jpg)
Increasing Demand for 3D Content
New driver: Artificial Intelligence (“Graphics for AI”)
3D Scene Semantic Segments
![Page 14: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/14.jpg)
Increasing Demand for 3D Content
New driver: Artificial Intelligence (“Graphics for AI”)
![Page 15: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/15.jpg)
Increasing Demand for 3D Content
New driver: Artificial Intelligence (“Graphics for AI”)
Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019
![Page 16: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/16.jpg)
Current Practice Can’t Meet Demand
Mannual 3D modeling: still slow, still hard to learn
Maya Solidworks
![Page 17: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/17.jpg)
Current Practice Can’t Meet Demand
Mannual 3D modeling: still slow, still hard to learn
Maya Solidworks
“The difficulty of generating images has been overwhelmed by a
five-thousand-fold improvement in price/performance of
computing.
What remains hard is modeling…the grand challenges in three-
dimensional graphics are to make simple modeling easy and to
make complex modeling accessible to far more people.”
— Bob Sproull, 1990
![Page 18: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/18.jpg)
Generative Models to the Rescue!?
For the purposes of this talk:
Generative model: a procedure which can be executed to generate novel instances of some 3D object class
![Page 19: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/19.jpg)
Benefits of Generative Models
3D content generation at scale
SpeedTree, Unreal Engine CityEngine
![Page 20: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/20.jpg)
Benefits of Generative Models
Explore modeling possibilities
Learning Implicit Fields for Generative Shape Modeling , Chen & Zhang 2019
![Page 21: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/21.jpg)
Benefits of Generative Models
Strong prior for vision systems
StructureNet: Hierarchical Graph Networks for 3D Shape Generation, Mo et al. 2019
![Page 22: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/22.jpg)
Two Classes of Generative Model
Procedural Models
Pros:
• High quality output by construction
Advanced Procedural Modeling of Architecture, Schwartz & Muller 2015
![Page 23: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/23.jpg)
Two Classes of Generative Model
Procedural Models
Pros:
• High quality output by construction
• Interpretable & editable
Advanced Procedural Modeling of Architecture, Schwartz & Muller 2015
![Page 24: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/24.jpg)
Two Classes of Generative Model
Procedural Models
Pros:
• High quality output by construction
• Interpretable & editable
Cons:
• Difficult to author
Advanced Procedural Modeling of Architecture, Schwartz & Muller 2015
![Page 25: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/25.jpg)
Two Classes of Generative Model
Procedural Models
Pros:
• High quality output by construction
• Interpretable & editable
Cons:
• Difficult to author
• Limited output variety
Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019
![Page 26: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/26.jpg)
Two Classes of Generative Model
Deep Generative Models
Pros:
• Variety (any class of shape)
• Easy to author (“just add data”)Learning Implicit Fields for Generative Shape Modeling , Chen & Zhang 2019
![Page 27: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/27.jpg)
Recent High-Profile Successes
3D-GAN Octree Generating Nets PointFlow
AtlasNet Pixel2Mesh IM-Net
![Page 28: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/28.jpg)
Two Classes of Generative Model
Deep Generative Models
Pros:
• Variety (any class of shape)
• Easy to author (“just add data”)
Cons:
• Inconsistent output quality
• Inscrutable representation
![Page 29: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/29.jpg)
Two Classes of Generative Model
Procedural Models
Pros:
• High quality output by construction
• Interpretable & editable
Cons:
• Difficult to author
• Limited output variety
Deep Generative Models
Pros:
• Variety (any class of shape)
• Easy to author (“just add data”)
Cons:
• Inconsistent output quality
• Inscrutable representation
How can we get all of these...
![Page 30: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/30.jpg)
Two Classes of Generative Model
Procedural Models
Pros:
• High quality output by construction
• Interpretable & editable
Cons:
• Difficult to author
• Limited output variety
Deep Generative Models
Pros:
• Variety (any class of shape)
• Easy to author (“just add data”)
Cons:
• Inconsistent output quality
• Inscrutable representation
...with none of these?
How can we get all of these...
![Page 31: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/31.jpg)
Generative Models Capture Variaton
Some modes can easily be expressed symbolically:
• Hierarchy
StructureNet: Hierarchical Graph Networks for 3D Shape Generation, Mo et al. 2019
![Page 32: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/32.jpg)
Generative Models Capture Variaton
Some modes can easily be expressed symbolically:
• Hierarchy
• Connectivity
GRASS: Generative Recursive Autoencoders for Shape Structures, Li et al. 2018
![Page 33: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/33.jpg)
Generative Models Capture Variaton
Some modes can easily be expressed symbolically:
• Hierarchy
• Connectivity
• Symmetry
• ...
GRASS: Generative Recursive Autoencoders for Shape Structures, Li et al. 2018
![Page 34: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/34.jpg)
Generative Models Capture Variaton
Some modes are hard to express symbolically:
• Fine-detailed geometry
Learning Implicit Fields for Generative Shape Modeling , Chen & Zhang 2019
![Page 35: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/35.jpg)
Generative Models Capture Variaton
Some modes are hard to express symbolically:
• Fine-detailed geometry
• Complex inter-part correlations
• ...Learning Implicit Fields for Generative Shape Modeling , Chen & Zhang 2019
![Page 36: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/36.jpg)
Generative Models Capture Variaton
Some modes can easily be expressed symbolically:
• Hierarchy
• Connectivity
• Symmetry
• ...
Some modes are hard to express symbolically:
• Fine-detailed geometry
• Complex inter-part correlations
• ...
Design Philosophy:
Use symbols where possible
Use neural nets for everything else
![Page 37: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/37.jpg)
Neurosymbolic 3D Model:
A generative model of a class of 3D objects which models some modes of variability via explicit symbols and others via a neural latent space
![Page 38: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/38.jpg)
Neurosymbolic 3D Model Design Space
Neurosymbolic
models of shape
structure
This talk
![Page 39: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/39.jpg)
NEUROSYMBOLIC MODELS OF SHAPE STRUCTURE
![Page 40: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/40.jpg)
What Do I Mean by Shape Structure?
• Parts (as oriented bounding boxes)
• Relations
• Hierarchy, connectivity, symmetry, ...
• Useful despite low geometric detail
• Ex: robot motion planning infer all
parts + relations given point cloud
observation
• Focus on manufactured objects
• E.g. chairs, tables, airplanes...
StructureNet: Hierarchical Graph Networks for 3D Shape Generation, Mo et al. 2019
![Page 41: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/41.jpg)
What Do I Mean by Shape Structure?
• Parts (as oriented bounding boxes)
• Relations
• Hierarchy, connectivity, symmetry, ...
• Useful despite low geometric detail
• Ex: robot motion planning infer all
parts + relations given point cloud
observation
• Focus on manufactured objects
• E.g. chairs, tables, airplanes...
• Can extend to organic objects via e.g.
generalized cylinder decompositionGeneralized Cylinder Decomposition, Zhou et al. 2015
![Page 42: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/42.jpg)
The “Holy Grail” of Structure Modeling
A single, interpretable procedural model that generates the structures of every object in a given shape class (e.g. chairs, airplanes)
But...
![Page 43: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/43.jpg)
Two Classes of Generative Model
Procedural Models
Pros:
• High quality output by construction
• Interpretable & editable
Cons:
• Difficult to author
• Limited output variety
Can a strategic use of neural nets eliminate these?
![Page 44: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/44.jpg)
Eliminating Procedural Cons
Problem: Hard to author
Solution: Train a neural net to write them for us
Problem: Limited output variety
Solution: Latent space of neural net will capture the variability that the symbolic program does not
![Page 45: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/45.jpg)
[SIGGRAPH Asia 2020]
![Page 46: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/46.jpg)
A Neurosymbolic 3D Modeling Pipeline
![Page 47: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/47.jpg)
ShapeAssemblyAn “assembly language” for part-based shapes
Low-level instructions Operates by assembling parts
![Page 48: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/48.jpg)
Anatomy of a ShapeAssembly Program
![Page 49: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/49.jpg)
Execution Semantics
![Page 50: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/50.jpg)
Execution Semantics
Semantics of attach
![Page 51: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/51.jpg)
Execution Semantics
![Page 52: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/52.jpg)
Execution Semantics
![Page 53: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/53.jpg)
Execution Semantics
Macros:
squeeze, reflect, translate
expand into multiple Cuboid +
attach statements
![Page 54: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/54.jpg)
Execution Semantics
Differentiable execution:
Output geometry is
differentiable with respect to
continuous parameters of input
program
![Page 55: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/55.jpg)
A Neurosymbolic 3D Modeling Pipeline
![Page 56: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/56.jpg)
A Neurosymbolic 3D Modeling Pipeline
![Page 57: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/57.jpg)
Extracting Programs from Shapes
Local region of an input hierarchical part graph
“Chair back”
“Chair back side bars”
“Chair back center slats”
![Page 58: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/58.jpg)
Extracting Programs from Shapes
Locally flattening the hierarchy to make interacting leaf parts siblings
![Page 59: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/59.jpg)
Extracting Programs from Shapes
Shortening leaf parts that intersect other leaf parts
![Page 60: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/60.jpg)
Extracting Programs from Shapes
Locating attachment points between parts
![Page 61: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/61.jpg)
Extracting Programs from Shapes
Forming leaf parts into symmetry groups
![Page 62: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/62.jpg)
Extracting Programs from Shapes
Ordering Attachments
• Due to imperative semantics,
attach order matters
• Heuristics to prune possible
orders, then check which one
produces output that best
fits the shape
![Page 63: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/63.jpg)
A Neurosymbolic 3D Modeling Pipeline
![Page 64: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/64.jpg)
A Neurosymbolic 3D Modeling Pipeline
![Page 65: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/65.jpg)
Learning to Write ShapeAssembly Programs
![Page 66: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/66.jpg)
Learning to Write ShapeAssembly Programs
![Page 67: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/67.jpg)
WHAT CAN YOU DO WITH IT?
![Page 68: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/68.jpg)
Novel Shape Generation
![Page 69: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/69.jpg)
Novel Shape Generation
![Page 70: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/70.jpg)
Novel Shape Generation
![Page 71: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/71.jpg)
Editing Generated Programs
![Page 72: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/72.jpg)
Editing Generated Programs
![Page 73: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/73.jpg)
Comparison Conditions
• 3D PRNN:Sequence of boxes, but no hierarchy or relations
• StructureNet:Hierarchy of boxes w/ symmetry relations, but no explicit parametric attachments
![Page 74: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/74.jpg)
Ours Generates Better Novel Shapes
![Page 75: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/75.jpg)
Ours Generates Better Novel Shapes
![Page 76: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/76.jpg)
Ours Generates Better Novel Shapes
Ours are also quantifiably more compact, physically stable, and distributionally similar to a held-out validation set
![Page 77: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/77.jpg)
Ours Produces Better Interpolation
Our interpolations are quantifiably smoother,in terms of both structure and geometry
![Page 78: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/78.jpg)
Point Cloud “Parsing”
![Page 79: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/79.jpg)
Point Cloud “Parsing”
![Page 80: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/80.jpg)
Point Cloud “Parsing”
![Page 81: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/81.jpg)
WHAT’S NEXT?
![Page 82: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/82.jpg)
Neurosymbolic 3D Model Design Space
Neurosymbolic
models of shape
structure
This talk
![Page 83: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/83.jpg)
Neurosymbolic 3D Model Design Space
Neurosymbolic
models of shape
structure Neurosymbolic
models of part
geometry
Neurosymbolic models
of surface appearance
(e.g. texture)
Neurosymbolic
models of part
functionalities
Joint generative models which
couple all of these
![Page 84: Neurosymbolic 3D Models...Learning to Generalize Kinematic Models to Novel Objects, Abbatematteo et al. 2019 Current Practice Can’t Meet Demand Mannual 3D modeling: still slow, still](https://reader036.vdocuments.us/reader036/viewer/2022071005/5fc2cf17ff7c3739d640585a/html5/thumbnails/84.jpg)
THANKS!