MASC: Multi-scale Affinity with Sparse Convolutionfor 3D Instance Segmentation

Technical Report

Chen LiuWashington University in St. Louis

chenliu@wustl.edu

Yasutaka FurukawaSimon Fraser University

furukawa@sfu.ca

Abstract

We propose a new approach for 3D instance segmenta-tion based on sparse convolution and point affinity predic-tion, which indicates the likelihood of two points belong-ing to the same instance. The proposed network, built uponsubmanifold sparse convolution [3], processes a voxelizedpoint cloud and predicts semantic scores for each occupiedvoxel as well as the affinity between neighboring voxelsat different scales. A simple yet effective clustering algo-rithm segments points into instances based on the predictedaffinity and the mesh topology. The semantic for each in-stance is determined by the semantic prediction. Experi-ments show that our method outperforms the state-of-the-art instance segmentation methods by a large margin on thewidely used ScanNet benchmark [2]. We share our codepublicly at https://github.com/art-programmer/MASC.

1. Introduction

The vision community has witnessed tremendousprogress in 3D data capturing and processing techniquesin recent years. Consumer-grade depth sensors enable re-searchers to collect large-scale datasets of 3D scenes [2,1, 7]. The emergence of such datasets empowers learning-based methods to tackle a variety of 3D tasks, such as objectrecognition, part segmentation and semantic segmentation.Among them, 3D instance segmentation is an important yetchallenging task as it requires accurate segmentation of 3Dpoints based on instances without a fixed label set. In thispaper, we propose a simple yet effective method to learn theaffinity between points based on which points are clusteredinto instances.

The irregularity of 3D data poses a challenge for 3Dlearning techniques. Volumetric CNNs [21, 15, 12] are firstexplored as they are straightforward extensions of their suc-cessful 2D counterparts. Various techniques are proposed toreduce the cost of expensive 3D convolutions. OctNet [17]

and O-CNN [19] utilize the sparsity of 3D data via theoctree representations to save computation. Other meth-ods [9, 14] process 3D point cloud directly without vox-elization. While showing promising results, these methodslack the ability of modeling local structures of the inputpoint cloud. Later methods [16, 8, 10] model local depen-dencies with various tricks, but the number of points pro-cessed by the network is still quite limited (e.g., 4, 096). Apoint cloud of an indoor space usually contains much morenumber of points, which means the network can only pro-cess a slice of the input point cloud at each time, which dis-ables global reasoning of the space. Recently, Graham etal. [3] propose an super-efficient volumetric CNN basedon submanifold sparse convolution [4] to process the entirepoint cloud of an indoor scene, which achieves promisingresults on the semantic segmentation task. In this paper, weadopt sparse convolution and propose a clustering algorithmbased on learned multi-scale affinities to tackle the 3D in-stance segmentation problem.

Instance segmentation is more challenging than seman-tic segmentation as it requires the additional reasoning ofobjects. Instance segmentation methods can be categorizedinto two groups, proposal-based approaches and proposal-free approaches. Proposal-based approaches builds sys-tem upon object detection and append segmentation mod-ules after bounding box proposals. Inspired by the recentsuccess of Mask R-CNN [5] on 2D instance segmenta-tion, 3D-SIS [6] develops a proposal-based system whichachieves the state-of-the-art performance on 3D instancesegmentation evaluated using the ScanNet benchmark [2].GSPN [22] presents a generative model for generating pro-posals. FrustumNet [13] un-project 2D proposals to 3Dspace for the segmentation network to process. On the otherhand, proposal-free methods cluster points into instancesbased on the similarity metrics. SGPN [20] trains a networkto predict semantic labels, a similarity matrix between allpairs of points, and point-wise confidence for being a seedpoint, from which instance segmentation results are gener-ated. However, the similarity between most pairs of points

1

arX

iv:1

902.

0447

8v1

[cs

.CV

] 1

2 Fe

b 20

19

Figure 1. We use a U-Net architecture with submanifold sparse convolutions [3] to process the entire point cloud of an indoor scene andpredict semantic scores for each point as well as the affinity between neighboring voxels at different scales. A simple yet effective clusteringalgorithm groups points into instances based on the predicted affinity and the mesh topology.

is not informative and introduces unnecessary learning chal-lenge, limiting the network to process only 4, 096 points ateach time. We address this issue by predicting only simi-larity between neighboring points at multiple scales, as didin [11] for 2D instance segmentation, and develops a clus-tering algorithm to group points based on the learned localsimilarity and the input mesh topology.

2. Methods

As shown in Fig. 1, we use the same U-Net architec-ture [18] with submanifold sparse convolution used in [3]for semantic segmentation. The sparse U-Net first voxelizedthe entire point cloud from each ScanNet scene, with eachpoint represented by its coordinate, color, and local surfacenormal. Following [3], we set the voxel size as 2cm×2cm×2cm and use 4, 096×4, 096×4, 096 voxel grids so that theentire scene can be voxelized sparsely and each voxel usu-ally contains at most 1 point. After voxelization, submani-fold sparse convolution layers with skip connections gener-ate feature maps of different spatial resolutions. We appenda semantic branch with one fully connected layer to predictpoint-wise semantics and add multiple affinity branches atdifferent scales to predict the similarity scores between anactive voxel and its 6 neighbors. We denote the finest scale,which has 4, 096 × 4, 096 × 4, 096 voxels, as scale 0 andscale s has resolution 4,096

s2 ×4,096s2 ×

4,096s2 .

We treat the input mesh as the initial graph (V,E),and propose a clustering algorithm to group points into in-stances based on the multi-scale affinity field and the se-mantic prediction. Note that the clustering is conductedon the mesh graph while the network predicts the affinitybetween neighboring voxels with fixed neighborhood. Sowe first compute the affinity between two notes by takingthe average affinity between neighboring voxel pairs con-

Algorithm 1 The clustering algorithm based on multi-scaleaffinityRequire: Voxel affinity As(p, q) at each scale, input mesh(V,E)return Clustering result for Vrepeat

Initialize A(Vi, Vj)if (Vi, Vj) ∈ E then

A(Vi, Vj)← avgs(avgp∈Vi,q∈Vj(As(p, q)))

elseA(Vi, Vj)← 0

end ifMap Vi to a neighbor M(Vi) with high similarityif maxVj (A(Vi, Vj)) > 0.5 thenM(Vi)← argmaxVj

(A(Vi, Vj))else

M(Vi)← Vi

end ifAssign Vi to cluster Ck if M(Vi) 6= Vi,M(Vi) ∈ Ck

Update E ← {(Ck, Cl)|∃Vi, Vj : Vi ∈ Ck, Vj ∈Cl, (Vi, Vj) ∈ E}Update V ← C

until The update does not change anything

necting two nodes. At the beginning, each node containsonly one point in the original point cloud which occupiesone voxel and the affinity between two nodes is simply theaffinity between corresponding voxels. As the node grows,a node could occupy multiple voxels in the finest scale, andif no less than 4s points of the node fall inside the voxel ata higher scale s, we say that the node occupies this voxel.With node affinities, we map each node Vi to its most simi-lar neighbor M(Vi) if the similarity between them is higher

2

than 0.5, and to itself otherwise. With the mapping, we cancluster nodes into groups such that every node in a groupis mapped to another node in the same group and none ofthe node in a group is mapped to other groups. We treateach cluster as a new node and update edges correspond-ingly. We repeat the process until no change happens (i.e.,every node is mapped to itself). The above procedure issummarized in Alg. 1.

Compared against the clustering algorithm in [11] for 2Dinstance segmentation, our clustering algorithm is more ag-gressive as it merges nodes in parallel, and has the poten-tial of being implemented using GPU operations. After theclustering process, each instance takes the semantic labelwith the maximum votes from its points.

3. Results

3.1. Implementation details

We implement the proposed method using PyTorchbased on the code of [3]1. Besides voxelizing and aug-menting ScanNet scenes in the same way of [3], we addpoint normals to the input and use instance labels to gen-erate affinity supervision between neighboring voxels. Twoneighboring voxels at scale s have a similarity score 1 if 1)each of them contains at least 4s points and 2) their instancedistributions among points are the same. To better predictlocal affinity at regions where the input mesh is sparse, wefurther augment the input mesh by randomly sampling 5points inside each triangle which spans at least 2 voxel ateither dimension. An edge is added between two sampledpoints if they are voxelized into neighboring voxels at thefinest scale, and a sampled point is also connected with itsclosest vertex of the original triangle. This random den-sification also enriches the training data. We use 2 scalesfor the affinity prediction. We train the network on a Titan1080ti GPU for 20 epochs.

3.2. Quantitative evaluation

We evaluate the instance segmentation performance onthe ScanNet benchmark [2]. To measure the confidence ofeach instance for the evaluation purpose, we collect all pre-dicted instances in the training set and train a smaller net-work to predict if an instance is valid for the predicted la-bel. The network encodes the point cloud of the instanceusing sparse convolutions and encodes the predicted labelwith a fully connected layer. We concatenate two featuresand add two fully connected layers to predict the final con-fidence score. In addition, we find that the clustering al-gorithm sometimes struggles to segment objects which areco-planar with other larger objects. So we add additional in-stances for semantic classes which are often planar, namely

1https://github.com/facebookresearch/SparseConvNet

pictures, curtains, shower curtains, sinks, and bathtubs (usu-ally contain only the bottom surfaces), by finding connectedcomponents of such classes based on the predicted seman-tics. Our method out-performs the state-of-the-art methodsby a large margin. Table 1 shows the comparison againstpublished results (with citable publications).

3.3. Qualitative evaluation

We show qualitative results for ScanNet scenes in thevalidation set in Fig. 2.

Figure 2. Qualitative results for ScanNet scenes. In each row, weshow the input mesh on the left and our instance segmentationresult on the right.

4. DiscussionThough this simple method already achieves promising

results, several improvements are required to further boostits performance. First, the clustering algorithm is currentlyimplemented sequentially and thus slow. The algorithmis parallel in theory and it is possible to implement it onGPU and extend it to enable back-propagation for end-to-end training. Besides the speed issue, the effect of multi-scale is under-explored. In practice, we find using 2 scales isfast to train and achieves good performance but it is unclearthe role played by each scale. With more exploration on themulti-scale affinity, it is possible to design a better cluster-ing algorithm which uses affinity of more scales to achievebetter performance. Finally, the current method sometimesfails to distinguish co-planar objects.

3

Table 1. Instance segmentation evaluation on the ScanNet benchmark (AP with IOU threshold 0.5)Method avg bath. bed book. cabi. chair coun. curt. desk door other pict. refr. show. sink sofa table toil. wind.

3D-SIS [6] 0.382 1.000 0.432 0.245 0.190 0.577 0.013 0.263 0.033 0.320 0.240 0.075 0.422 0.857 0.117 0.699 0.271 0.883 0.235

GSPN [22] 0.306 0.500 0.405 0.311 0.348 0.589 0.054 0.068 0.126 0.283 0.290 0.028 0.219 0.214 0.331 0.396 0.275 0.821 0.245

SGPN [20] 0.143 0.208 0.390 0.169 0.065 0.275 0.029 0.069 0.000 0.087 0.043 0.014 0.027 0.000 0.112 0.351 0.168 0.438 0.138

Ours 0.447 0.528 0.555 0.381 0.382 0.633 0.002 0.509 0.260 0.361 0.432 0.327 0.451 0.571 0.367 0.639 0.386 0.980 0.276

5. AcknowledgementThis research is partially supported by National Science

Foundation under grant IIS 1618685, Natural Sciences andEngineering Research Council under grants RGPAS-2018-528049, RGPIN-2018-04197, and DGDND-2018-00006,and Google Faculty Research Award. We thank Nvidia fora generous GPU donation.

References[1] I. Armeni, S. Sax, A. R. Zamir, and S. Savarese. Joint 2d-3d-

semantic data for indoor scene understanding. arXiv preprintarXiv:1702.01105, 2017.

[2] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser,and M. Nießner. Scannet: Richly-annotated 3d reconstruc-tions of indoor scenes. In Proc. IEEE Conf. on ComputerVision and Pattern Recognition (CVPR), volume 1, page 1,2017.

[3] B. Graham, M. Engelcke, and L. van der Maaten. 3d seman-tic segmentation with submanifold sparse convolutional net-works. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 9224–9232, 2018.

[4] B. Graham and L. van der Maaten. Submanifold sparseconvolutional networks. arXiv preprint arXiv:1706.01307,2017.

[5] K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask r-cnn.In Computer Vision (ICCV), 2017 IEEE International Con-ference on, pages 2980–2988. IEEE, 2017.

[6] J. Hou, A. Dai, and M. Nießner. 3d-sis: 3d seman-tic instance segmentation of rgb-d scans. arXiv preprintarXiv:1812.07003, 2018.

[7] B.-S. Hua, Q.-H. Pham, D. T. Nguyen, M.-K. Tran, L.-F. Yu,and S.-K. Yeung. Scenenn: A scene meshes dataset withannotations. In 3D Vision (3DV), 2016 Fourth InternationalConference on, pages 92–101. IEEE, 2016.

[8] Q. Huang, W. Wang, and U. Neumann. Recurrent slice net-works for 3d segmentation of point clouds. In Proceedingsof the IEEE Conference on Computer Vision and PatternRecognition, pages 2626–2635, 2018.

[9] R. Klokov and V. Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. InComputer Vision (ICCV), 2017 IEEE International Confer-ence on, pages 863–872. IEEE, 2017.

[10] C. Liu, J. Wu, and Y. Furukawa. Floornet: A unified frame-work for floorplan reconstruction from 3d scans. arXivpreprint arXiv:1804.00090, 2018.

[11] Y. Liu, S. Yang, B. Li, W. Zhou, J. Xu, H. Li, and Y. Lu.Affinity derivation and graph merge for instance segmenta-

tion. In Proceedings of the European Conference on Com-puter Vision (ECCV), pages 686–703, 2018.

[12] D. Maturana and S. Scherer. Voxnet: A 3d convolutionalneural network for real-time object recognition. In IntelligentRobots and Systems (IROS), 2015 IEEE/RSJ InternationalConference on, pages 922–928. IEEE, 2015.

[13] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas. Frustumpointnets for 3d object detection from rgb-d data. In Pro-ceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 918–927, 2018.

[14] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deeplearning on point sets for 3d classification and segmentation.Proc. Computer Vision and Pattern Recognition (CVPR),IEEE, 1(2):4, 2017.

[15] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. J.Guibas. Volumetric and multi-view cnns for object classifi-cation on 3d data. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 5648–5656,2016.

[16] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hi-erarchical feature learning on point sets in a metric space. InAdvances in Neural Information Processing Systems, pages5099–5108, 2017.

[17] G. Riegler, A. Osman Ulusoy, and A. Geiger. Octnet: Learn-ing deep 3d representations at high resolutions. In Proceed-ings of the IEEE Conference on Computer Vision and PatternRecognition, pages 3577–3586, 2017.

[18] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convo-lutional networks for biomedical image segmentation. InInternational Conference on Medical image computing andcomputer-assisted intervention, pages 234–241. Springer,2015.

[19] P.-S. Wang, Y. Liu, Y.-X. Guo, C.-Y. Sun, and X. Tong.O-cnn: Octree-based convolutional neural networks for 3dshape analysis. ACM Transactions on Graphics (TOG),36(4):72, 2017.

[20] W. Wang, R. Yu, Q. Huang, and U. Neumann. Sgpn: Similar-ity group proposal network for 3d point cloud instance seg-mentation. In Proceedings of the IEEE Conference on Com-puter Vision and Pattern Recognition, pages 2569–2578,2018.

[21] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, andJ. Xiao. 3d shapenets: A deep representation for volumetricshapes. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 1912–1920, 2015.

[22] L. Yi, W. Zhao, H. Wang, M. Sung, and L. Guibas. Gspn:Generative shape proposal network for 3d instance segmen-tation in point cloud. arXiv preprint arXiv:1812.03320,2018.

4