3d shape segmentation with projective convolutional networkskalo/papers/shapepfcn/shapepfcn... ·...
TRANSCRIPT
![Page 1: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/1.jpg)
3D Shape Segmentation with Projective Convolutional Networks
Evangelos Kalogerakis1 Melinos Averkiou2 Subhransu Maji1 Siddhartha Chaudhuri31University of Massachusetts Amherst 2University of Cyprus 3IIT Bombay
![Page 2: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/2.jpg)
Goal: learn to segment & label parts in 3D shapes
3D Geometric representation
![Page 3: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/3.jpg)
Goal: learn to segment & label parts in 3D shapes
3D Geometric representation
ShapePFCN
![Page 4: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/4.jpg)
Goal: learn to segment & label parts in 3D shapes
3D Geometric representation
Labeled3D shape
backseatbasearmrest
ShapePFCN
![Page 5: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/5.jpg)
Goal: learn to segment & label parts in 3D shapes
...Training Dataset
3D Geometric representation
Labeled3D shape
backseatbasearmrest
ShapePFCN
![Page 6: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/6.jpg)
Motivation: Parsing RGBD data
backseatbasearmrest
...
RGBD data from “A Large Dataset of Object Scans”Choi, Zhou, Miller, Koltun 2016
Our result
![Page 7: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/7.jpg)
Simari, Nowrouzezahrai, Kalogerakis, Singh, SGP 2009
Motivation: 3D Modeling & Animation
EarHeadTorsoBackUpper arm
Upper leg
FootTail
Lower armHand
Lower leg
Kalogerakis, Hertzmann, Singh, SIGGRAPH 20103D Models
Animation Texturing
![Page 8: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/8.jpg)
Related WorkTrain classifiers on hand‐engineered descriptorse.g., Kalogerakis et al. 2010, Guo et al. 2015
curvature shape contexts geodesic dist.
![Page 9: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/9.jpg)
Related WorkTrain classifiers on hand‐engineered descriptorse.g., Kalogerakis et al. 2010, Guo et al. 2015
Concurrent approaches:Volumetric / octree‐based methods: Riegler et al. 2017 (OctNet), Wang et al. 2017 (O‐CNN), Klokov et al. 2017 (kd‐net)
Point‐based networks: Qi et al. 2017 (PointNet / PointNet++)
Graph‐based / spectral networks: Yi et al. 2017 (SyncSpecCNN)
Surface embedding networks:Maron et al. 2017
curvature shape contexts geodesic dist.
![Page 10: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/10.jpg)
Related WorkTrain classifiers on hand‐engineered descriptorse.g., Kalogerakis et al. 2010, Guo et al. 2015
Concurrent approaches:Volumetric / octree‐based methods: Riegler et al. 2017 (OctNet), Wang et al. 2017 (O‐CNN), Klokov et al. 2017 (kd‐net)
Point‐based networks: Qi et al. 2017 (PointNet / PointNet++)
Graph‐based / spectral networks: Yi et al. 2017 (SyncSpecCNN)
Surface embedding networks:Maron et al. 2017
Our method:view‐basednetwork
curvature shape contexts geodesic dist.
![Page 11: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/11.jpg)
Key Observations3D models are often designed for viewing.
![Page 12: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/12.jpg)
Key Observations3D models are often designed for viewing.
![Page 13: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/13.jpg)
Key Observations
Empty inside!
3D models are often designed for viewing.
![Page 14: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/14.jpg)
Key Observations
+ noise, missing regions etc
3D scans capture the surface.
![Page 15: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/15.jpg)
Key Observations
Parts do not touch!
(not easily noticeable to the viewer, yet geometric implications on topology, connectedness...)
3D models are often designed for viewing.
![Page 16: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/16.jpg)
Key ObservationsShape renderings can be treated as photos of objects (without texture)
Shape renderings can be processed by powerful image‐based architectures through transfer learning from massive image datasets.
Image‐basednetwork
Chair!
Airplane!
(Su, Maji, Kalogerakis, Learned‐Miller, ICCV 2015)
![Page 17: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/17.jpg)
Deep architecture that combines view‐based convnets for part reasoning on rendered shape images & prob. graphical models for surface processing.
Key Idea
![Page 18: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/18.jpg)
Deep architecture that combines view‐based convnets for part reasoning on rendered shape images & prob. graphical models for surface processing.
Key challenges:‐ Select views to avoid surface information loss & deal with occlusions
Key Idea
![Page 19: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/19.jpg)
Deep architecture that combines view‐based convnets for part reasoning on rendered shape images & prob. graphical models for surface processing.
Key challenges:‐ Select views to avoid surface information loss & deal with occlusions‐ Promote invariance under 3D shape rotations
Key Idea
![Page 20: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/20.jpg)
Deep architecture that combines view‐based convnets for part reasoning on rendered shape images & prob. graphical models for surface processing.
Key challenges:‐ Select views to avoid surface information loss & deal with occlusions‐ Promote invariance under 3D shape rotations‐ Joint reasoning about parts across multiple views + surface
Key Idea
![Page 21: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/21.jpg)
fuselagewingvert. stabilizerhoriz. stabilizer
Viewselection
ShapePFCN
Pipeline
![Page 22: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/22.jpg)
ShapePFCNViewselection
fuselagewingvert. stabilizerhoriz. stabilizer
Pipeline
![Page 23: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/23.jpg)
Input: shape as a collection of rendered viewsFor each input shape, infer a set of viewpoints that maximally cover its surface across multiple distances.
![Page 24: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/24.jpg)
Render shaded images (normal dot view vector) encoding surface normals.
Shaded images
Input: shape as a collection of rendered views
![Page 25: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/25.jpg)
Render also depth images encoding surface position relative to the camera.
Shaded images
Depthimages
Input: shape as a collection of rendered views
![Page 26: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/26.jpg)
Perform in‐plane camera rotations for rotational invariance.
Input: shape as a collection of rendered views
Shaded images
Depthimages
0o, 90o, 180o, 270orotations
![Page 27: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/27.jpg)
Projective convnet architecture
Surfaceconfidence
maps
Each pair of depth & shaded images is processed by a FCN.Views are not ordered (no view correspondence across shapes).
FCN
FCN
FCN
Shaded images
Depthimages
[Long, Shelhamer, and Darrell 2015]
![Page 28: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/28.jpg)
Projective convnet architecture
Surfaceconfidence
maps
Each pair of depth & shaded images is processed by a FCN.Views are not ordered (no view correspondence across shapes).
FCN
FCN
FCN
shared filtersShaded images
Depthimages
[Long, Shelhamer, and Darrell 2015]
![Page 29: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/29.jpg)
The output of each FCN branch is a view‐based confidence map per part label.
Projective convnet architecture
Surfaceconfidence
maps
hor. stabilizer
FCN
FCN
FCN
Shaded images
Depthimages View‐based mapshared filters
![Page 30: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/30.jpg)
Projective convnet architecture
wing
FCN
FCN
FCN
Shaded images
Depthimages
The output of each FCN branch is a view‐based confidence map per part label.
View‐based mapshared filters
![Page 31: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/31.jpg)
Aggregate & project the image confidence maps from all views on the surface.
Projective convnet architecture
FCN
FCN
FCN
Shaded images
Depthimages
Surface‐based map
View‐based mapshared filters
![Page 32: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/32.jpg)
FCN
FCN
FCN
Shaded images
Depthimages
For each surface element (triangle), find all pixels that include it in all views. Surface confidence: use max of these pixel confidences per label.
Image2Surface projection layer
max
View‐based mapshared filters
Surface‐based map
![Page 33: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/33.jpg)
FCN
FCN
FCN
Shaded images
Depthimages
For each surface element (triangle), find all pixels that include it in all views. Surface confidence: use max of these pixel confidences per label.
Image2Surface projection layer
max
View‐based mapshared filters
Surface‐based map
![Page 34: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/34.jpg)
FCN
FCN
FCN
Shaded images
Depthimages
For each surface element (triangle), find all pixels that include it in all views. Surface confidence: use max of these pixel confidences per label.
Image2Surface projection layer
max
View‐based mapshared filters
Surface‐based map
![Page 35: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/35.jpg)
CRF layer for spatially coherent labelingLast layer performs inference in a probabilistic model defined on the surface.
R1
R2
R3
R4
R1, R2, R3, R4…random variablestaking values:
fuselagewingvert. stabilizerhoriz. stabilizer
![Page 36: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/36.jpg)
CRF layer for spatially coherent labelingConditional Random Field: unary factors based on surface‐based confidences
11 2 4 '
.. , '3, , , ... 1( | ) ( | ) ( , | )
f n f ff f fR R R R RP P P
ZR R
shape views surface
Unary factors(FCN confidences)
R1
R2
R3
R4
![Page 37: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/37.jpg)
11 2 4 '
.. , '3, , , ... 1( | ) ( | ) ( , | )
f n f ff f fR R R R RP P P
ZR R
shape views surface
Projective convnet architecture: CRF layerPairwise terms favor same label for triangles with:(a) similar surface normals(b) small geodesic distance
Pairwise factors(geodesic+normal distance)
R1R3
R4R2
![Page 38: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/38.jpg)
11 2 4 '
.. , '3, , , ... 1( | ) ( | ) ( , | )
f n f ff f fR R R R RP P P
ZR R
shape views surfacemax
Projective convnet architecture: CRF layerInfer most likely joint assignment to all surface random variables (mean‐field)
MAP assignment (mean–field inference)
R1R3
R4R2
![Page 39: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/39.jpg)
Forward passinference (convnet+CRF)
FCN
CRF
![Page 40: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/40.jpg)
TrainingThe architecture is trained end‐to‐end with analytic gradients.
FCN
Backpropagation / joint training (convnet+CRF)
CRF
![Page 41: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/41.jpg)
TrainingThe architecture is trained end‐to‐end with analytic gradients.Training starts from a pretrained image‐based net (VGG16), then fine‐tune.
Pre‐trainedFCN
Backpropagation / joint training (convnet+CRF)
CRF
![Page 42: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/42.jpg)
Dataset used in experimentsEvaluation on ShapeNet + LPSB + COSEG (46 classes of shapes)50% used for training / 50% used for test split per Shapenet categoryMax 250 shapes for training. No assumption on shape orientation.
[Yi et al. 2016]
![Page 43: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/43.jpg)
Dataset used in experimentsEvaluation on ShapeNet + LPSB + COSEG (46 classes of shapes)50% used for training / 50% used for test split per Shapenet categoryMax 250 shapes for training. No assumption on shape orientation.
[Yi et al. 2016]
![Page 44: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/44.jpg)
Results
ShapeBoost Guo et al. ShapePFCN81.2 80.6 87.5
Labeling accuracy on ShapeNet test dataset:
![Page 45: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/45.jpg)
Results
ShapeBoost Guo et al. ShapePFCN81.2 80.6 87.576.8 76.8 84.7
Ignore easy classes(2 or 3 part labels)
8% improvement in labeling accuracy for complex categories (vehicles, furniture)
Labeling accuracy on ShapeNet test dataset:
![Page 46: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/46.jpg)
Results
ShapeBoost Guo et al. ShapePFCN81.2 80.6 87.576.8 76.8 84.7
Labeling accuracy on ShapeNet test dataset:
Ignore easy classes(2 or 3 part labels)
8% improvement in labeling accuracy for complex categories (vehicles, furniture)
Labeling accuracy on LPSB+COSEG test dataset:ShapeBoost Guo et al. ShapePFCN
84.2 82.1 92.2
![Page 47: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/47.jpg)
“ground-truth” ShapeBoost ShapePFCN
![Page 48: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/48.jpg)
ShapeBoost ShapePFCN
Object scans from “A Large Dataset of Object Scans” Choi et al. 2016
![Page 49: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/49.jpg)
Summary
• Deep architecture combining view‐based FCN & surface‐based CRF
• Multi‐scale view selection to avoid loss of surface information
• Transfer learning frommassive image datasets
• Robust to geometric representation artifacts
Acknowledgements: NSF (CHS‐1422441, CHS‐1617333, IIS‐ 1617917)Experiments were performed in the UMass GPU cluster (400 GPUs!)obtained under a grant by the MassTech Collaborative
![Page 50: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/50.jpg)
Project page: http://people.cs.umass.edu/~kalo/papers/shapepfcn/
![Page 51: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/51.jpg)
Auxiliary slides
![Page 52: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/52.jpg)
conv4
conv5
fc6
What are the filters doing?Activated in the presence of certain patterns of surface patches
![Page 53: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/53.jpg)
conv4
conv5
fc6
What are the filters doing?Activated in the presence of certain patterns of surface patches
![Page 54: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/54.jpg)
conv4
conv5
fc6
What are the filters doing?Activated in the presence of certain patterns of surface patches
![Page 55: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/55.jpg)
Input: shape as a collection of rendered viewsFor each input shape, infer a set of viewpoints that maximally cover its surface across multiple distances.
![Page 56: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/56.jpg)
Input: shape as a collection of rendered viewsFor each input shape, infer a set of viewpoints that maximally cover its surface across multiple distances.
![Page 57: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/57.jpg)
Input: shape as a collection of rendered viewsFor each input shape, infer a set of viewpoints that maximally cover its surface across multiple distances.
![Page 58: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/58.jpg)
Input: shape as a collection of rendered viewsFor each input shape, infer a set of viewpoints that maximally cover its surface across multiple distances.
![Page 59: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/59.jpg)
Input: shape as a collection of rendered viewsFor each input shape, infer a set of viewpoints that maximally cover its surface across multiple distances.
![Page 60: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/60.jpg)
Input: shape as a collection of rendered viewsFor each input shape, infer a set of viewpoints that maximally cover its surface across multiple distances.
![Page 61: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/61.jpg)
FCN
FCN
FCN
Shaded images
Depthimages
max
View‐based mapshared filters
Surface‐based map
![Page 62: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/62.jpg)
TrainingThe architecture is trained end‐to‐end with analytic gradients.
FCN
Backpropagation / joint training (convnet+CRF)
CRF
![Page 63: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/63.jpg)
Challenges• 3D models have missing or non‐photorealistic texture
![Page 64: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/64.jpg)
Challenges• 3D models have missing or non‐photorealistic texture (focus on shape instead)
![Page 65: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/65.jpg)
ShapeNetCore: 8% improvement in labeling accuracyfor complex categories (vehicles, furniture etc)
![Page 66: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/66.jpg)
![Page 67: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/67.jpg)
3D models have arbitrary orientation “in the wild”.
Consistent shape orientation only in specific, well‐engineered datasets(often with manual intervention, no perfect alignment algorithm)
Key Observations
![Page 68: 3D Shape Segmentation with Projective Convolutional Networkskalo/papers/shapepfcn/ShapePFCN... · 3D Shape Segmentation with Projective Convolutional Networks ... Motivation: 3D Modeling](https://reader030.vdocuments.us/reader030/viewer/2022040409/5ec67985db0d1917dc6269e3/html5/thumbnails/68.jpg)
Comparisons pitfalls
• Method A assumes consistent shape alignment (or upright orientation), method B doesn’t. • Well, you may get much better numbers for B by changing its input!
• Convnet A has orders of magnitude more parameters than Convnet B.• It might also be easy to get better numbers for B, if you increase its number of filters!
• Convnet A has an architectural “trick” that Convnet B could also have (e.g., U‐net, ensemble).• Why not apply the same trick to B?
• Methods are largely tested on training data because of duplicate or near‐duplicate shapes.• The more you overfit, the better! Ouch!• 3DShapeNet has many identical models, or models with tiny differences (e.g., same airplane with different rockets)...