A Search-Classify Approach for Cluttered Indoor Scene Understanding

Liangliang Nan1, Ke Xie1, Andrei Sharf2

1 SIAT, China

2 Ben Gurion University, Israel

Digitalization of indoor scenes

Indoor scenes from Google 3D Warehouse

Acquisition of indoor scenes

Goal

• Scene understanding

Challenges

• Clutter– Densely populated– Arbitrary arrangements

• Partial representation– Occlusions

• Complex geometry

Classification & Segmentation

• Two interleaved problems– What are the objects?– Where are the objects?

• Chicken-egg problem– Classification needs segmentation– Segmentation needs a prior

Our solution

• Search – Propagate / accumulate patches

• Classify– Query classifier to detect object

Related Work

• Indoor scenes (This Session)– [Fisher et al. 2012] [Shao et al. 2012] [Kim et al. 2012]

• Semantic relationship– [Fisher et al. 2010, 2011]

• Recognition using depth + texture (RGB-D)– [Quigley et al.2009], [Lai and Fox 2010]

• Outdoor classification– [Golovinskiy et al. 2009]

• Semantic labeling– [Koppula et al. 2011]

Controlled region growing process

Our search-classify idea

0.6 0.8

0.92 0.94 0.94 0.94

Method overview

Training

Search-Classify

Point cloud features

– Height-size ratio of BBox– Aspect ratio of each layer– Bottom-top, mid-top size ratio– Change in COM along horizontal slabs

Bh

BdBw

Classifier

• Handle missing data– Occlusion

• Random decision forest– Efficient multi-class classifier

• Trained with both scanned and synthetic data– Manually segmented and labeled– 510 chairs – 250 tables – 110 cabinets – 40 monitors etc.

[Shotton et al. 2008, 2011]

Search-Classify

• Starts from seeds– Random patch triplets– Remove seeds with low confidence

• Accumulating neighbor patches– Highest classification confidence

• Stop condition– Steep decrease in classification confidence

0.65 0.92 0.93 0.88

Seed

• Segmented - classified objects problems– Overlap, outliers, ambiguities etc.

• Refinement – Outliers = patches with large distance

Segmentation refinement by template fitting

Template deformation

• Different styles for each class• Predefined scalable parts• Templates can deform [Xu et al. 2010]

Template deformation

• Different styles for each class• Predefined scalable parts• Templates can deform [Xu et al. 2010]

Fitting via template deformation

Confidence Fitting error Best fitting

• Best matching template– One-side Euclidean distance from points to template

Results and discussion

Results and discussion

Results and discussion

• Scalability test with varied object density

0 (25) 1 (45) 5 (60)

Results and discussion

• ComparisonLai et al. 2011

Ours

Limitation

• Upward assumption– Features– Template fitting

Future work

• Contextual information

Thank you