Introduction of Saliency Map

Presenter: Chien-Chi ChenAdvisor: Jian-Jiun Ding

1

Outline• Introduction of saliency map• Button-up approach

– L. Itti’s approach– Frequency-tuned– Multi-scale contrast– Depth of field– Spectral Residual approach– Global contrast based

• Top-down approach– Context-aware

• Information maximum– Measuring visual saliency by site entropy rate

2

Outline• Introduction of saliency map• Button-up approach

– L. Itti’s approach– Frequency-tuned– Multi-scale contrast– Depth of field– Spectral Residual approach– Global contrast based

• Top-down approach– Context-aware

• Information maximum– Measuring visual saliency by site entropy rate

3

Introduction of saliency map

• Low-level(contrast)– Color– Orientation– Size– Motion– Depth

• High-level– People– Context

Important!

4Judd et

al, 2009

Low-level

With face

detection

Outline• Introduction of saliency map• Button-up approach

– L. Itti’s approach– Frequency-tuned– Multi-scale contrast– Depth of field– Spectral Residual approach– Global contrast based

• Top-down approach– Context-aware

• Information maximum– Measuring visual saliency by site entropy rate

5

L. Itti’s approach

• Architecture:Gaussian Pyramids

R,G,B,Y Gabor pyramids for = {0º, 45º, 90º, 135º}

L. Itti’s approach• Center-surround Difference• Achieve center-surround difference through across-scale difference

• Operated denoted by Interpolation to finer scale and point-to-point subtraction

• One pyramid for each channel: I(), R(), G(), B(), Y()where [0..8] is the scale

L. Itti’s approach• Center-surround Difference

– Intensity Feature Maps• I(c, s) = | I(c) I(s)|• c {2, 3, 4}• s = c + where {3, 4}• So I(2, 5) = | I(2) I(5)|

I(2, 6) = | I(2) I(6)| I(3, 6) = | I(3) I(6)| …

• 6 Feature Maps

L. Itti’s approach• Center-surround Difference

•Color Feature Maps

Red-Green and Yellow-Blue

Center-surround DifferenceOrientation Feature Maps

•

+R-G

+R-G+G-R

+G-R +B-Y

+Y-B

+Y-B

+B-Y

+B-Y

Same c and s as with intensity

),(),(),,( sOcOscO

RG(c, s) = | (R(c) - G(c)) (G(s) - R(s)) |BY(c, s) = | (B(c) - Y(c)) (Y(s) - B(s)) |

L. Itti’s approach• Normalization Operator• Promotes maps with few strong peaks• Surpresses maps with many comparable

peaks1. Normalization of map to range [0…M]2. Compute average m of all local maxima 3. Find the global maximum M4. Multiply the map by (M – m)2

L. Itti’s approach

Inhibition of return

Example of Operation:

Frequency-tuned

12

Image Average

Gaussian blur

L

I a

b

( , )hc

hc hc

hc

L

I x y a

b

( , ) ( , )hc

S x y I I x y

Multi-scale contrast

• Saliency algorithm

Image Saliency map

Multi-scale contrast

Center-surround histogram

Color spatial-distribution

ConditionalRandomField

13

Multi-scale contrast

Multi-scale contrast• Local summation of

laplacian pyramid

Center-surround histogram• Distance between histograms

of RGB color:

2

1 ( )( , ) || ( ) ( ) ||

L l lc

l x N xf x I I x I x

22 ( )1

( , )2 ( )

i is

s i is

R RR R

R R

* 2

( )( ) arg max ( ( ), ( ))s

R xR x R x R x

*

2 * *

{ | ( )}( , ) ( ( ), ( ))h xx s

x x R xf x I R x R x

14

Multi-scale contrast

• Color spatial-distribution

Image(RGB)

GMMDistance from pixel x to image center

The variance of Coordinate of pixel x and y

( , ) ( | ) (1 ( )) (1 ( ))s xc

f x I p c I V c D c

15

Multi-scale contrast

• Energy term:

• Saliency object:

1 ,( | ) ( , ) ( , , )

Kk k x x x

x k x xE A I F a I S a a I

( , ), 0( , )

1 ( , ), 1k x

k xk x

f x I aF a I

f x I a

• Pairwise feature:

,( , , ) | | exp( )x x x x x xS a a I a a d

, || ||, 2x x x xd I I L norm

2 1(2 || || )x xI I

16

Multi-scale contrast

• CRF:

• The derivative of the log-likelihood with respect to

1( | ) exp( ( | ))P A I E A I

Z

* arg max log ( | ; )n n

nP A I

k

17

Depth of field

• As the spread of single lens reflex camera, more and more low depth of field(DOF) images are captured.

• However, current saliency detection methods work poorly for the low DOF images.

18

Depth of field

• Algorithm:

19

Depth of field

• Classification: • Focal Point: In a low DOF image

DOGRectangle with the highest edge density, and center is initial focal point

2( , ) ( , )

d

S i j S i j Ae

• Composition Analysis:segmentation Region

1 2 3

r

i

A n d

A mr rS S e

20

Spectral Residual Approach

• First scaling image to 64x64.• Then we smoothed the saliency map with a

gaussian filter g(x) ( = 8).

21

Global contrast-based

• Histogram based contrast(Lab):

2( )O N 2( ) ( )O N O n

Quantization of Lab

Each channel to have 12 different value

312 1728

8522

Global contrast-based

• Region based contrast– Segment the Image– [Efficient graph-based image segmentation]

23

Outline• Introduction of saliency map• Button-up approach

– L. Itti’s approach– Frequency-tuned– Center-surround– Depth of field– Spectral Residual approach– Global contrast based

• Top-down approach– Context-aware

• Information maximum– Measuring visual saliency by site entropy rate

24

Context-Aware

• Goal: Convey the image content

25

Liu et al, 2007

Context-Aware

• Distance between a pair of patches:

( , )( , )

1 ( , )color i j

i jposition i j

d p pd p p

c d p p

salient

High j

Context-Aware

• Distance between a pair of patches:

High for K most similar

Saliency

k

rq K most similar patches at scale r1

11 exp ( , )

i j

Kr r ri

k

S d p qK

Context-Aware

• Salient at:– Multiple scales foreground– Few scales background

1

1 Mrr

i ir r

S SM

Scale 1 Scale 4

Context-Aware

• Foci =

• Include distance map

0.8iS

1 ( )focid i

X

iS

ˆ 1 ( )i i fociS S d i

Outline• Introduction of saliency map• Button-up approach

– L. Itti’s approach– Frequency-tuned– Center-surround– Depth of field– Spectral Residual approach– Global contrast based

• Top-down approach– Context-aware

• Information maximum– Measuring visual saliency by site entropy rate

30

Measuring visual saliency by site entropy rate

31

1

Measuring visual saliency by site entropy rate

32

A fully-connected graph representation is adopted for each

2

Sub-band graph representation

33

Sub-band graph representation

34

Measuring visual saliency by site entropy rate

35

A random walk is adopted on each sub-band graph. And Site entropy rate(SER) is measured the average information from a node to the other

3

The site entropy rate

•

•

36

ijij

ijj

P

, :, ,

2i

i i ij ijj i j j i

WW W

W

Conclusion

• Image processing is funny • Unusual in its neighborhood will correspond

to high saliency weight• Contrast is the key of saliency

37

Reference[1] R. Achanta, F. Estrada, P. Wils, and S. S¨usstrunk. Salient region detection and

segmentation. In ICVS, pages 66–75. Springer, 2008. 410, 412, 414[2] R. Achanta, S. Hemami, F. Estrada, and S. S¨usstrunk. Frequency-tuned salient

region detection. In CVPR, pages 1597–1604, 2009. 409, 410, 412, 413, 414, 415[3] L. Itti, C. Koch, and E. Niebur. A model of saliency based visual attention for rapid

scene analysis. IEEE TPAMI, 20(11):1254–1259, 1998. 409, 410, 412, 414[4] X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In CVPR,

pages 1–8, 2007. 410, 412, 413, 414[5] S. Goferman, L. Zelnik-Manor, and A. Tal. Context-aware saliency detection. In

CVPR, 2010. 410, 412, 413, 414, 415[6] MM Cheng, GX Zhang, N. J. Mitra, X. Huang, S.M. Hu. Global Contrast based Salient

Region Detect. In CVPR, 2011 .[7] T. Liu, Z. Yuan, J. Sun, J.Wang, N. Zheng, T. X., and S. H.Y. Learning to detect a

salient object. IEEE TPAMI, 33(2):353–367, 2011. 410[8] W. Wang, Y. Wang, Q. Huang, W. Gao, Measuring Visaul Saliency by Site Entropy

Rate, In CVPR, 2010.

38