RGB-(D) Scene Labeling: Features and Algorithms

Ahmed TahaMay 2014

Supervised by Dr : Marwan A. Torki

Introduction Scene labeling challenges Pipeline

Feature Extraction Super-pixel formulation and classification Classifying segmentation tree paths Classifying super-pixels MRF

Datasets and results

Agenda

Scene Labeling Labeling of each pixel in an image to a certain

class Scene Labeling can be done

Indoors Label a Sofa in a Bedroom Label a door in a living room

Outdoors Label a car in street Label building in street

Scene Labeling

Scene Labeling

Scene Labeling

Indoor scene labeling challenges Large variations of scene types Lack of distinctive features Poor illumination

Scene Labeling

Benefits of using depth feature in scene labeling Increased accuracy and robustness Body pose estimation 3D mapping Object recognition 3D modeling and interaction

Scene Labeling

Pipeline

1. Extract features using Kernel descriptor (KDES).

2. Aggregate descriptors in dense region into super-pixels using Efficient match kernels (EMK)

3. Classify super-pixels using Linear support vector machine (SVM)

4. Label super-pixels by classifying paths of segmentation tree.

5. Label super-pixels using super-pixel MRF

Pipeline

Kernel Descriptors (KDES), a unified framework that uses different aspects of similarity (kernel) to derive patch descriptors. Image gradient Spin/normal Color Depth gradient

Features Extraction (Step 1)

Efficient match kernels (EMK) to transform and aggregate descriptors in a set S (grid locations in the interior of a superpixel ‘s’).

Super-pixels are not of the same size.

Super-pixel formation (Step 2)

Linear Support vector machine (SVM) Non-probabilistic binary linear classifier.

Classify superpixels (Step 3)

Classifying paths in segmentation tree

Contextual Models (Step 4)

Classifying paths in segmentation tree

Contextual Models (Step 4)

Classifying paths in segmentation tree

Contextual Models

Classifying paths in segmentation tree If we accumulate features over paths, the

accuracy continues to increase to the top level The initial part of the curves overlap,

suggesting there is little benefit going to superpixels at too fine scales

Contextual Models

Superpixel MRF with gPb

Contextual Models (Step 5)

Superpixel MRF with gPb standard MRF formulation. We use Graph Cut

to find the labeling that mini- mizes the energy of a pairwise MRF

Contextual Models (Step 5)

Pipeline

NYU-D dataset Improve accuracy from 50% to 76%

Stanford Background dataset Improve accuracy 79% to 82%

Datasets - Results

Datasets - Results

Rgb-(d) scene labeling: Features and algorithms X Ren, L Bo, D Fox - Computer Vision and

Pattern Recognition 2012 - ieeexplore.ieee.org Context by region ancestry

JJ Lim, P Arbeláez, C Gu, J Malik - Computer Vision, 2009 IEEE 2009 - ieeexplore.ieee.org

References