from interactive to semantic image segmentation
DESCRIPTION
From Interactive to Semantic Image Segmentation. Varun Gulshan Supervisors: Prof. Andrew Blake Prof. Andrew Zisserman. 20 Jan 2012. Two segmentation tasks. sky. background. building. tree. tree. object. person. car. car. road. bench. Interactive segmentation. - PowerPoint PPT PresentationTRANSCRIPT
From Interactive to SemanticImage Segmentation
Varun Gulshan
Supervisors:Prof. Andrew Blake Prof. Andrew Zisserman
20 Jan 2012
Two segmentation tasks
car
car
building
person
treetree
sky
roadbench
object
background
Interactive segmentation Semantic segmentation
Interactive segmentation Semantic segmentation
Thesis Flow
Chapter 3:Texture Features
Low level cues
Chapter 4:Star
convexity
Mid level cues
Chapter 5:Segmenting
humans
Bounding box interaction +
Top down cues
Chapter 6:Superpixel
based classification
Fully automatic segmentation
Chapter 3: Features for interactive segmentation
Low level texture features for improving interactive segmentation methods.
Texture features
Pure Texture Feature (L-shape):
Texture + Gray Feature (L-shape):
Texture features
Pure Texture Feature (Plus-shape):
Texture + Gray Feature (Plus-shape):
Camouflage Image Dataset
Introduced a dataset of 50 Camouflage images, to demonstrate the power of texture features.
Quantitative evaluation
+14% +21%
+7%+4%
Gra
yR
GB
Huge gain in accuracy obtained using texture features on top of gray scale images. Significant improvement on top of RGB images.
Chapter 4: Star Convexity and Extensions
Mid level shape constraints for reducing user effort in interactive segmentation systems.
Chapter 4: Star convexity
SingleStar
MultipleStars
GeodesicStar
Robot user evaluation
Updated segmentation
Biggest connected component
Initial brush strokes Segmentation output with current
interaction
Error segmentation
False positive
False negative
New Brush Stroke
Centre of connected component
New Brush Stroke
New brush stroke placed
Process is repeated upto 20 strokes
Segmentation after 20 strokes
Robot user evaluation
Method SP-IG SP-SIG SP-LIG BJ RW GSCseqEffort 17.78 15.77 15.14 12.35 12.31 9.63
Our method takes least effort
Chapter 5: Learning to segment humans
Using top down cues to segment specific object categories.
Segmenting humans
Bounding box (given/detected)
Top down HOG prediction Bottom up refinement
Kinect Data Acquisition
RGB image Kinect scene labels Cleaned up Ground truth
Dataset of roughly 3500 images acquired using the Kinect
Top down learning
Local Image Local HOG Local mask
Classifier trained to predict segmentation masks for local windows based on their HOG descriptor.
Bottom up refinement
Top down segmentation
Local Color model window
Local color model unaries
Final segmentation
…..
Chapter 6: Semantic segmentation
Fully automatic segmentation based upon learning from multiple superpixelisations.
Combing multiple superpixelisations
Various methods to learn from multiple superpixelisations explored: 1. Avg-Indep 2. Avg-Union 3. LPβ-Indep 4. IofR-Joint
GlobalPb Veksler QuickShift
Quantitative evaluation
+7% +5% +6% +3%
Sin
gle
supe
rpix
elis
atio
nM
ultip
le
supe
rpix
elis
atio
ns
Combining multiple superpixelisations improves performance.
Novel pairwise features
....
....
CRF trained jointly for appearance and novel pairwise features.
Over to you!