22nd Computer Vision Winter WorkshopNicole M. Artner, Ines Janusch, Walter G. Kropatsch (eds.)Retz, Austria, February 6–8, 2017

Semantic segmentation for room categorization

Ajda Lampe, Luka Cehovin, Rok Mandeljc, Matej KristanFaculty of computer and information science, University of Ljubljana

Vecna pot 113, Ljubljana, Sloveniaajda.lampe@gmail.com {luka.cehovin,rok.mandeljc,matej.kristan}@fri.uni-lj.si

Abstract. Room category recognition is a highlydesired functionality in household robots as it en-ables natural indoor navigation without having torely solely on range sensors. Significant research hasbeen invested into the problem of image-based roomrecognition. All approaches assume that the imagecontains a single category and usually apply holis-tic scene models. This assumption is often violatedin realistic robotic scenarios, in which a robot ob-serves several room categories within its view. Wepropose an alternative approach for room categoryrecognition, based on per-pixel semantic segmenta-tion. Our approach decomposes the input image intosemantic regions, and easily deals with images con-taining either a single or multiple categories. Asa side product, each room category is localized inthe image, which makes our approach suitable forrobotic exploration and navigation based on visualfeedbacks. Our approach is evaluated on a newdataset, annotated with pixel-wise ground-truth la-bels for eight room categories. Results show that ourapproach outperforms the state-of-the-art in recogni-tion as well as pixel-wise localization. Our approachexhibits a remarkable robustness to rotation and out-performs the holistic state-of-the-art approach evenon examples with a single category.

1. Introduction

Visual room categorization is a high-level com-puter vision task in which the system is presentedwith an image of an indoor scene, and is expectedto predict the most likely semantic label of the roomdepicted in the image. Such systems are extremelyuseful in mobile robotics, where the maps of envi-ronment, used for navigation, do not include suchsemantic information. A typical household robotwould need to recognize the semantic meaning of

Figure 1: Example of an image with multiple roomcategories present. Holistic approaches, such asPlacesNet [23], can only report a single (dominant)room category, while our approach produces a coarsesegmentation that enables multi-category recognitionas well as in-image category localization.

its current location in order to successfully reasonabout the given tasks, and to be able to navigate thesurroundings and perform operations in appropriaterooms.

Recently, computer-vision-based room classifica-tion methods have achieved high classification ac-curacy by employing convolutional neural networks,trained on large annotated datasets of indoor and out-door scenes [23, 6, 21]. However, the main limita-tion of these approaches lies in the formalization ofthe classification task, where the image is classifiedholistically, i.e., the entire scene is assigned a singlesemantic label. This is true even for methods thatotherwise utilize spatial information [9, 13]. In real-istic scenarios, the observed indoor scenes frequentlycontain multiple room categories. Figure 1 shows anexample with different rooms and areas with distinctsemantic functionality present in a single view. In

such cases, the majority of existing room classifica-tion approaches is limited to producing only a singleoutput label. Some of the approaches do provide aprobability distribution over the possible room cate-gories, but this is still insufficient for accurate cate-gory localization within the robot field of view.

We propose a change of paradigm in room cate-gorization by casting it as a problem of approximatesemantic segmentation. Our approach uses local andcontextual information to infer the most likely roomcategory at each pixel. The final room category isestimated by aggregating per-pixel classifications. Incontrast to existing approaches that apply holistic ag-gregation into a single category, our approach detectsclusters of category labels in the image and can beconsidered as non-parametric estimation. Thus theapproach is expected to be more robust than relatedapproaches in cases when observing an image with asingle as well as multiple categories. As a side prod-uct, our approach allows category localization withinan image.

We claim two contributions. The first contributionis a novel approach for room classification and local-ization, based on semantic segmentation, which wecall the RoomNet. To the best of our knowledge,this is the first method that addresses the problemof room classification through semantic segmenta-tion. Our second contribution is a new dataset forroom classification. In contrast to existing datasets,our dataset contains images with a single and multi-ple categories and is per-pixel labeled with the cat-egory identities. Our RoomNet is evaluated exten-sively on this dataset. Results conclusively showthat the RoomNet outperforms a recent state-of-the-art room classification approach by approximately15% on multiple as well as single category images,is more robust to view rotation and performs accuratelocalization.

The remainder of this paper is organized as fol-lows. Section 2 overviews the related work, Sec-tion 3 describes the proposed room categorizationapproach, Section 4 presents the experimental eval-uation, and conclusions are drawn in Section 5.

2. Related work

Majority of existing literature in the field ofvision-based room categorization employs holisticdescriptors; they construct a feature representationcorresponding to the entire image, and classify itwith a multi-class classifier. The traditional ap-

proaches rely on bag-of-words representations, con-structed from low-level features. In [3], authors con-struct a “bag of keypoints” representation from key-points, detected using Harris affine detector and en-coded by SIFT descriptor; the resulting feature vec-tor is classified using naive Bayes classifier. In [8],the holistic bag-of-words descriptor is constructedfrom HOG features, classified using an SVM. In [1],authors use SIFT features, compressed with LDA,and classify them using several machine learningtechniques. Texture descriptors, such as oriented tex-ture curves [19] and semantic proto-concepts [10]have also been adopted for general scene classifica-tion. To include spatial information in the holisticdescriptors, several researchers apply spatial pyramidhistograms [17, 14].

The second line of work employs object detec-tors and reasons about the room category based onoccurrence of their responses. In [9], authors pro-pose to process an input image with a large bank ofpre-trained object detectors, and combine their re-sponses into a single scene descriptor. The same fil-ter bank is used by [11], who construct a high-levelimage representation based on co-occurrences of de-tector responses. Such approaches largely depend onthe quality of used detectors, as well as their repre-sentativity for the particular room category. A moregeneral approach based on local description was pre-sented in [13]. It exploits both local and global vi-sual information by describing the scene using mul-tiple regions in a star-shaped model. Authors in [12]jointly learn the set of prototype regions and imageclassifiers by random sampling image regions; the fi-nal holistic representation is formed via concatena-tion of the obtained region responses. [15] train in-dividual object detectors and object-group detectors,and max-pool their responses in a spatial pyramid.In a similar scheme, [5] discover visually-coherentpatch clusters from an image collection that are max-imally discriminative with respect to the labels.

The turning point in room classification hasbeen the advent of large domain-specific evaluationdatasets, which have provided a platform for methodcomparison and further development. Datasets suchas MIT Indoor [13], SUN [20], Places205 [23], andPlaces365 [22] are among the most widely used.Some of them extend beyond indoor room categories.However, due to the holistic ground-truth labels, theyare suitable only for evaluation of holistic room clas-sification methods.

Figure 2: Schematic representation of the RoomNet network architecture.

The availability of large training datasets hasalso boosted the utilization of deep-learning tech-niques. In particular, convolutional neural networkshave been successfully applied to scene recognitionin [23]. The authors train the network on combi-nation of ImageNet [4] and Places205 [23] datasets,and use it as a generic feature extractor in combina-tion with an SVM to achieve state-of-the-art resultson several scene recognition datasets. In [6], authorsuse cutouts at multiple scales as the input to networksin order to better handle geometric invariance. Thesystem proposed in [21] generates activation mapsthat correspond to regions with discriminative infor-mation for each class, and uses them to classify im-ages. Another part-based system [18] uses CNN fea-tures extracted from generic region proposals to in-crease robustness to partial occlusions. In contrast toour work, all these methods still address the holistictask of predicting a single label for the input image.

The use of segmentation in our work is reminis-cent of the scene-parsing task [24]. The latter islow-level, and is concerned with segmenting the im-age into regions that correspond to individual objects.Therefore, our work can be seen as situated betweenthe high-level holistic scene classification and low-level scene parsing, and instead of providing a singlecategory label or segmented individual objects, aimsto segment image regions that correspond to distinctsemantic room types.

3. Methodology

The theoretical framework of our approach is theDeepLab [2] CNN architecture which was developedfor generic semantic segmentation. The DeepLabnetwork is based on the VGG-16 network [16], andis fine-tuned for segmentation of the 21 categoriesfrom Pascal VOC 2012 dataset. The bottom ten lay-

ers are based on VGG-16, comprising five series ofconvolutional layers with interspersed max-poolinglayers. At the top of the network, there are threefully-connected layers, separated by two drop-outlayers. The output of the final layer is a per-pixel la-bel distribution map, which is spatially interpolatedwith factor 8 to the original image resolution. TheDeepLab [2] originally applies a conditional randomfield (CRF) to improve per-pixel segmentation accu-racy.

Our network, which we call the RoomNet, is de-picted in Figure 2. The RoomNet re-uses the bot-tom ten convolutional and max-pool layers of theDeepLab network, and re-trains the top three fully-connected layers on the task of segmentation forroom localization and categorization. The top fully-connected (output) layer, l8, contains nine channelscorresponding to the eight room categories in ourdataset plus the background label. The ninth channelcorresponding to the background class is used onlyin the training phase to account for the "unassigned"pixels in the ground truth. The network architectureand parameters of individual layers are summarizedin Table 1.

3.1. Training

In the training stage, each spatial unit (collectionof the channel responses at the same spatial position)is connected to a softmax classifier that predicts themost likely class label at that location. Due to reduc-tions in the lower network layers, the resulting outputmap is subsampled by factor 8 compared to the inputimage size. The network upper layers are optimizedwith respect to the loss function, based on the sumof cross-entropy terms for each spatial unit, with allpositions and labels being weighted equally, and thetargets being the pixel-wise ground-truth labels (also

Table 1: Properties of the RoomNet network layers.The li layers are convolutional, pi layers are max-pooling, and di layers are drop-out layers. In theupper part of the network, the fully-connected lay-ers (l6, l7, and l8) are implemented via convolutionallayers. The last column contains filter parameters:kernel size K, padding P , and stride S. If not speci-fied, P = 0 and S = 1. H denotes the parameter ofthe “atrous algorithm” introduced by [2].

Layer Size Channels Filter parameters

l1 256× 256 64 K = 3, P = 1p1 129× 129 64 K = 2, P = 1, S = 2l2 129× 129 128 K = 3, P = 1p2 65× 65 128 K = 2, P = 1, S = 2l3 65× 65 256 K = 3, P = 1p3 33× 33 256 K = 2, P = 1, S = 2l4 33× 33 512 K = 3, P = 1p4 32× 32 512 K = 2, S = 1l5 32× 32 512 K = 3, P = 2, H = 2p5 32× 32 512 K = 3, P = 1, S = 1

l6 32× 32 4096 K = 4, P = 6, H = 4d1 32× 32 4096l7 32× 32 4096 K = 1d2 32× 32 4096

l8 32× 32 9 K = 1

subsampled by the factor 8). For details of the costfunction, we refer the interested reader to [2].

3.2. Testing

During the inference, the response maps of the l8layer are interpolated to the original image size. Af-ter the interpolation, the final segmentation map isobtained by selecting a label with highest probabilityat each pixel location. In contrast to the DeepLab [2],we do not apply the CRF, since high per-pixel accu-racy is not required for room category inference. Theboundaries between different room types in imagesare not as well defined as the boundaries between ob-jects, therefore each room category can be localizedonly approximately. Any further spatial regulariza-tion would merely result in an unnecessary increasein computational cost.

4. Experimental evaluation

The implementation details of our approach areprovided in Section 4.1. The dataset is described inSection 4.2. The experimental results are describedand discussed in Section 4.3.

4.1. Implementation details

The RoomNet network was implemented in theCaffe framework [7]. We have used the training por-tion of the dataset, presented in Section 4.2, to trainthe top three fully-connected layers. The training hy-perparameters were set to the values recommendedby the Caffe documentation. Layers l6 and l7 are ini-tialized with the DeepLab data and only fine-tunedfor our scenario with learning rate 10−3. Layer l8is initialized with random weights, and trained fromscratch with initial learning rate 10−2, which is after-wards decreasing with factor λ = 0.1 per epoch. Weuse mini-batch size of 10 images, training momen-tum µ = 0.9, and weight decay of 0.0005.

4.2. Dataset

The existing datasets are tailored for evaluation ofholistic room categorization performance and con-tain only a single category label per image. Wethus assembled a new dataset that provides pixel-wise room category annotations. Our dataset is basedon the MIT Indoor [13] dataset, which contains im-ages of 67 indoor scene categories. Note that we havenoticed that in fact several images from this datasetcontain multiple room categories, but are annotatedonly with a single category. We selected a subset of 8categories corresponding to most common householdrooms: bathroom, bedroom, children room, closet,corridor, dinning room, kitchen, and living room.This selection was extended by additional imagesfrom the internet, that contained multiple room cat-egories within a single image. To account for local-ization ambiguity, we introduced a special categoryfor the background, which accounts for the image re-gions that cannot be reliably attributed to any of theroom categories.

The dataset is summarized in Figure 3. It contains8029 images, 1778 of which are in the testing set.We have made sure that the distribution of classesin both training and testing sets are equal. In to-tal 465 images contain two or more room categories.Each image is manually annotated on per-pixel basiswith the nine possible category labels. From the co-occurrence matrix in Figure 3 we can see that mostcategories appear together at least in one image. Themost frequently co-occurring rooms are kitchen, din-ing room, and living room. Such arrangements aretypical for small apartments.

Figure 3: Our dataset overview. The pie-chart on theleft presents the distribution of room categories, theco-occurrence matrix on the right shows the numberof simultaneous appearances of two categories in thesame image.

4.3. Experimental results

We compare the proposed approach against thestate-of-the-art room categorization method calledthe PlacesNet from [23]. The PlacesNet applies apre-trained CNN as a generic feature extractor to en-code the input image as a 4096-dimensional featurevector and an SVM for classification. We considerthe variant with Places205-AlexNet network, whichwas trained on the Places205 [23] dataset. To trainthis pipeline on our dataset, we extract the CNNfeatures from training images with the pre-trainedPlaces205-AlexNet, and use them to train the lin-ear one-vs-rest SVM. Note that we train only withthe subset of training images that contain a singleroom category. In our preliminary study we trainedwith multi-room samples as well (i.e., treating eachsuch image as multiple samples with correspondinglabels), but have observed significantly worse perfor-mance.

Two evaluation scenarios are considered. The firstscenario considers room category presence detection.Here, the task is to correctly identify the labels for thecategories present in the image. The second scenarioconsiders localization of these categories as well. Inaddition we conduct a robustness study in which theinput images are rotated. Image rotation is com-mon in mobile platforms due to camera tilting duringrobot motion.

Table 2: Summary of the experimental results. Thetable shows the accuracy for single-room classifica-tion task, the best F-score for multi-room classifica-tion task, and the IoU score for the localization task.

Approach Accuracy F-score IoU

PlacesNet [23] 0.79 0.72 0.73RoomNet (ours) 0.90 0.83 0.84

4.4. Room category presence detection

To maintain relation to the existing holistic bench-marks, both methods were first compared in termsof classification accuracy on a subset of images thatonly contain a single room category.

The most likely category is determined in ourRoomNet by determining the category with maxi-mum number of pixels associated. The accuracy ofclassification is shown in the left-most column of Ta-ble 2. Our RooomNet significantly outperforms thePlacesNet even on this, traditionally holistic, task byapproximately 14 %. This boost speaks in favor ofour non-parametric formulation of category estima-tion through segmentation. The ambiguous pixels,or pixels stemming potentially from other categories,represent non-dominant categories and do not affectdominant class prediction, thus increasing the robust-ness of RoomNet.

Next, the entire dataset was considered for multi-category classification. Both methods were adaptedto report classes where the reported score of the classis higher than the Θ portion of the maximum outputclass. For PlacesNet the class score is the output ofeach one-vs-all SVM classifier (distance to the de-cision hyperplane) normalized by softmax to obtainthe presence probability of each class. For Room-Net the probability of each class corresponds to thepercentage of the pixels assigned to it.

The multi-category results are shown in Figure 4in terms of F-score and precision-recall with re-spect to the threshold Θ. The RoomNet consistentlyoutperforms the PlacesNet in F-score. In fact, theRoomNet achieves an excellent performance alreadyat moderate threshold levels. Since our dataset isdominated by single-category images, we show theF-scores separately for single-category and multiple-category images (second row in Figure 4). Thesingle-category graph is similar to the overall F-scoreplot. The multiple-category graphs shows that ourRoomNet outperforms the PlacesNet when a single

category is reported (Θ = 1). The improvementis most significant for Θ selected for each methodseparately. Table 2 depicts average performance ofRoomNet and PlacesNet at their best values of Θ onsingle and mixed room category images. The Room-Net outperforms the PlacesNet by over 15 %. Theprecision-recall curve of RoomNet in Figure 4 is con-sistently above that of the PlacesNet. The best recallachieved by PlacesNet is close to 0.8, with 0.68 pre-cision. Our RoomNet achieves approximately 0.84precision at the 0.8 recall, thus by far outperformingthe PlacesNet.

The results again support our non-parametric for-mulation in which the category of a single pixel doesnot pollute the global classification score, but rathercontributes to the specific class through a cluster ofpixels.

Figure 4: Top row: F-score and precision-recallcurves for multi-class categorization with respect toparameter Θ. Bottom row: F-score plots with respectto parameter Θ only for single room and multipleroom images.

4.5. Room category localization

The task of room category localization in animage is posed as a semantic segmentation prob-lem. Similarly to semantic segmentation evalua-tion, we estimate the localization performance bythe intersection-over-union (IoU) score. The score ismodified to account for the background class that wehave reserved for ambiguous regions in the groundtruth annotation. Pixels that are labeled as back-ground in the ground truth are ignored in the cal-culation of the IoU score since the correct class is

unknown at these locations.Holistic methods are unable to provide category

location information, since they provide only a sin-gle output label. Thus, the segmentation map forPlacesNet is computed by assigning the output labelto all pixels in the image. Such straightforward ex-tension indeed results in suboptimal performance onmulti-room images as only a single room category isreported. On the other hand, PlacesNet should haveadvantage on images that contain only a single class,since a correct global decision results in 100 % label-ing accuracy. The RoomNet is required to make suchpredictions for each pixel independently, which is acompetitive disadvantage in single-category images.This is illustrated in Figure 5. The PlacesNet methodpredicts more images with a perfect IoU score, butthose are all single-category images. It also receivesa very low score on a large portion of the testingdataset. These are the images with multiple roomsand mis-categorized single rooms where the IoU is0 %. The RoomNet method achieves high (althoughsometimes not perfect) IoU in single-category im-ages and achieves a better global performance thanthe PlacesNet. According to the average IoU scorein Table 2, the RoomNet outperforms the PlacesNetby 15 % in IoU score.

Figure 5: IoU score for all testing images sorted indecreasing order.

Figure 6 shows confusion matrices computed frompixel-wise classifications. Our RoomNet outper-forms PlacesNet across all categories. The Places-Net sometimes miss-classifies bathroom and chil-drens room for bedroom. The RoomNet significantlyreduces these mixups. The reduction in false classifi-cations is also apparent in the case of dinning rooms.PlacesNet often miss-classifies them as a kitchen ora living room. The miss-classification is reduced byRoomNet by 15 percentage points (∼20 % improve-ment).

4.6. Robustness to image rotation

To evaluate the rotational invariance we have re-peated the classification and localization experiments

Figure 6: Confusion matrices for localization task.

from Section 4.4 and Section 4.5 on a modifieddatasets where all images were rotated by an angleα ∈ [0, 180] degrees.

The results for the single and multiple room clas-sification are shown in Figure 8. Results show a re-markable robustness of RoomNet to rotations of upto 30 degrees. The drop in accuracy is within 10 %,and approximately 5 % in F-score and IoU. In caseof PlacesNet, the drop in accuracy is twice as large,while in F-score and IoU is four times as large. At30 degree rotation, the PlacesNet accuracy is approx-imately 0.59, while the accuracy of our RoomNet isstill at approximately 0.83. Thus the improvement ofour RoomNet over PlacesNet is in order of 40 % at30 degree rotation. For significant rotations, the pe-formance of both approaches drops, but the drop isstill more significant for the PlacesNet than for ourRoomNet.

The segmentation performance for different imagerotation angles is visualized in Figure 8. On singleroom images the holistic PlacesNet method has theadvantage as it estimates only global category, how-ever, if the method fails, the entire segmentation isincorrect. The RoomNet produces some small incor-rect regions in such cases because it has to classifythem individually, but overall the results are consis-tent with the ground truth labels. In case of multi-room images the difference is even more apparent.PlacesNet can only recognize a single category, usu-ally the dominant one. But in many cases the re-sults are completely incorrect. The RoomNet pro-vides a decent localization. For unusual rotation an-gles, the segmentation also includes small regionsof incorrect predictions that could be improved ina post-processing step which takes into account thesize of the regions and prior information about co-occurrence of room categories.

Figure 7: Classification accuracy for single roomclassification task and F-score for multi-room clas-sification in relation to input image rotation.

5. Conclusion

A new approach for image-based room catego-rization was proposed. In contrast to related ap-proaches that perform a global classification by ag-gregating features into a holistic representation, ourapproach applies a non-parametric technique. Thenon-parametric property comes from estimating thesupport for a particular category at a pixel level andseparately aggregating the votes for each category.This is achieved by casting the categorization prob-lem as a semantic segmentation task in which eachpixel is assigned a room category label. A CNNframework is used to construct our semantic segmen-tation network – the RoomNet. In addition to thenovel room recognition network we presented a newdataset for room categorization and localization. Thedataset is per-pixel annotated groundtruth with eightroom categories.

Extensive analysis was performed to comparethe RoomNet with a recent state-of-the-art Places-Net [23]. Our approach outperforms the Places-Net on the task of single-category as well as multi-category classification. The RoomNet exhibits an ex-cellent ability for localization of categories in the im-ages and is significantly more robust to image rota-tion than the PlacesNet. The RoomNet achieves ap-proximately 15 % better accuracy in room category

Figure 8: Examples of segmentation obtained fordifferent rotation angles. The top example shows per-formance for a single room image, while the othertwo examples show performance for images wheretwo rooms are present in the same image.

detection. Our RoomNet goes beyond the state-of-the-art by allowing room category localization in ad-

dition to recognition.Our future work will include extending the Room-

Net by increasing the (indoor and outdoor) place cat-egories. The approach will be extended to a mobilerobot scenario in which the provided segmentationsfrom RoomNet will be used to increase the accuracyof localization and motion planning during roboticexploration.

References[1] B. Ayers and M. Boutell. Home interior classifica-

tion using SIFT keypoint histograms. In 2007 IEEEConference on Computer Vision and Pattern Recog-nition, pages 1–6. IEEE, 2007. 2

[2] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy,and A. L. Yuille. Semantic image segmentation withdeep convolutional nets and fully connected CRFs.arXiv preprint arXiv:1412.7062, 2014. 3, 4

[3] G. Csurka, C. Dance, L. Fan, J. Willamowski, andC. Bray. Visual categorization with bags of key-points. In Workshop on statistical learning in com-puter vision, ECCV, volume 1, pages 1–2. Prague,2004. 2

[4] J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, andL. Fei-Fei. Imagenet: A large-scale hierarchical im-age database. In 2009 IEEE Conference on Com-puter Vision and Pattern Recognition (CVPR’09),pages 248–255, 2009. 3

[5] C. Doersch, A. Gupta, and A. A. Efros. Mid-level vi-sual element discovery as discriminative mode seek-ing. In C. J. C. Burges, L. Bottou, M. Welling,Z. Ghahramani, and K. Q. Weinberger, editors, Ad-vances in Neural Information Processing Systems26, pages 494–502, 2013. 2

[6] Y. Gong, L. Wang, R. Guo, and S. Lazebnik. Multi-scale orderless pooling of deep convolutional activa-tion features. In European Conference on ComputerVision, pages 392–407. Springer, 2014. 1, 3

[7] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev,J. Long, R. Girshick, S. Guadarrama, and T. Dar-rell. Caffe: Convolutional architecture for fast fea-ture embedding. In Proceedings of the 22nd ACMinternational conference on Multimedia, pages 675–678. ACM, 2014. 4

[8] M. Juneja, A. Vedaldi, C. V. Jawahar, and A. Zisser-man. Blocks that shout: Distinctive parts for sceneclassification. In Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition,pages 923–930, 2013. 2

[9] L.-J. Li, H. Su, L. Fei-Fei, and E. P. Xing. Objectbank: A high-level image representation for sceneclassification & semantic feature sparsification. InAdvances in neural information processing systems,pages 1378–1386, 2010. 1, 2

[10] R. Margolin, L. Zelnik-Manor, and A. Tal. OTC: Anovel local descriptor for scene classification. In Eu-ropean Conference on Computer Vision, pages 377–391. Springer, 2014. 2

[11] G. Mesnil, S. Rifai, A. Bordes, X. Glorot, Y. Ben-gio, and P. Vincent. Unsupervised learning of se-mantics of object detections for scene categoriza-tion. In International Conference on Pattern Recog-nition Applications and Methods (ICPRAM 2015),pages 209–224, 2015. 2

[12] S. N. Parizi, A. Vedaldi, A. Zisserman, andP. Felzenszwalb. Automatic discovery and optimiza-tion of parts for image classification. arXiv preprintarXiv:1412.6598, 2014. 2

[13] A. Quattoni and A. Torralba. Recognizing indoorscenes. In Computer Vision and Pattern Recogni-tion, 2009. CVPR 2009. IEEE Conference on, pages413–420. IEEE, 2009. 1, 2, 4

[14] F. Sadeghi and M. F. Tappen. Latent pyramidal re-gions for recognizing scenes. In European Confer-ence on Computer Vision, pages 228–241. Springer,2012. 2

[15] A. Sadovnik and T. Chen. Hierarchical object groupsfor scene classification. In 19th IEEE InternationalConference on Image Processing (ICIP 2012), pages1881–1884, 2012. 2

[16] K. Simonyan and A. Zisserman. Very deep convo-lutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014. 3

[17] S. Singh, A. Gupta, and A. A. Efros. Unsuper-vised discovery of mid-level discriminative patches.In European Conference on Computer Vision, pages73–86. Springer, 2012. 2

[18] P. Uršic, R. Mandeljc, A. Leonardis, and M. Kristan.Part-Based Room Categorization for Household Ser-vice Robots, 2016. 3

[19] J. C. van Gemert, J. Geusebroek, C. J. Veenman,C. G. M. Snoek, and A. W. M. Smeulders. Robustscene categorization by learning image statistics incontext. In 2006 Conference on Computer Visionand Pattern Recognition Workshop (CVPRW’06),page 105. IEEE, 2006. 2

[20] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Tor-ralba. Sun database: Large-scale scene recognitionfrom abbey to zoo. In Computer vision and pat-tern recognition (CVPR), 2010 IEEE conference on,pages 3485–3492. IEEE, 2010. 2

[21] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, andA. Torralba. Learning Deep Features for Discrimina-tive Localization. arXiv preprint arXiv:1512.04150,2015. 1, 3

[22] B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, andA. Oliva. Places: An image database for deep sceneunderstanding. arXiv preprint arXiv:1610.02055,2016. 2

[23] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, andA. Oliva. Learning deep features for scene recog-nition using places database. In Advances in neu-ral information processing systems, pages 487–495,2014. 1, 2, 3, 5, 7

[24] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Bar-riuso, and A. Torralba. Semantic understanding ofscenes through the ade20k dataset. arXiv preprintarXiv:1608.05442, 2016. 3