Vision and Image Processing Lab, Indian Institute of Technology Bombay, India
Avik Hati, Subhasis Chaudhuri, Rajbabu [email protected], [email protected], [email protected]
Saliency Detection based on a Sparsity of Segments Model
Introduction
• Saliency: measure of importance of objects in images
• Salient object
captures attention
distinctive in features
• Applications
Object segmentation
Video summarization
Content based image compression
Image and video quality assessment
• Only the important parts are processed
• Reduction in complexity
Typical Methods
• Local approach (center-surround window)
Pixel based [Harel et al., Itti et al.]
Boundary of large smooth salient object is detected
• Global approach (global comparison)
Superpixel based [Perazzi et al.]
Patch based [Margolin et al., Yan et al.]
Salient objects are not uniformly highlighted
• Frequency domain [Guo et al., Hou et al.]
Problem Definition
• Object based approach• Obtain a saliency map of an image such that
pixels within an object to have equal saliency salient objects to be extracted completely retain exact boundary of the salient object number of object segmentation to be small
Texture Removal
Gaussian Mixture Model
• EM algorithm assuming GMM• Minimize to find optimum number of Gaussians, K
: Gaussian pmfs: mixing proportions
• Number of segments is K
Input image Histogram Texture-removedimage
Histogram
TV : Total Variation: Weights: Texture-removed piecewise constant image: Regularization parameter
i
g
ke
i
( )if x
255
0 1
( ) ( )
arg min
k
k i i
x i
kk
e hist x f x
K e
Segmentation
• Obtain thresholds by minimizing probability of misclassification
, ,
Saliency Computation
• Color saliency
: area of region
• Spatial saliency
: spatial std. dev. of region
Salient Object Extraction
• Retain exact object boundaries
• Image Matting [Levin et al.]
• Eroded saliency output as foreground scribble
• Dilated saliency output as background scribble
Saliency Results
Challenges
• Choice of regularizer constant Large value removes small objects
• Texture-removed image may not always facilitate meaningful segmentation
Over smoothing Combination of Y, Cb, Cr
components
Proposed saliency maps in (e) are complete and without textures, holes, unlike (b-d). Comparison of precision, recall and F-measure of SF, RC, CA with the proposed method (Our) on the MSRA dataset. (see paper for details)
i
Texture-removed image Segmented regions (K=3)GMM (K=3)
1
01
( ) ( )i K
i
K
e i i i
i
P f x dx f x dx
0 0 255K
iA rir
Segmentationresult
Color saliencyoutput (thresholded)
Spatial saliency
Input image-2 Segmentation
Color saliency
1,2,...,
( ) 1max
j
s
j j K
s j
j
Inputimage
Saliencyoutput
Salient object
Inputimage
(a)
Chenget al.(b)
Perazziet al.(d)
Proposed(e)
Gofermanet al.
(c)
Multiple salient objects
Salient region that can be overlooked by presence of known objects
Very small salient objects
2( )
, ,
min i
i TVFgi Y Cb Cr
g I g
1
( ) ( ) ( , )K
c i c j i
i
s j r d r r
Precision-Recall
Salient object: Butterfly
Salient object: Green apple
jr
