cvpr2010: context-aware saliency detection

Olympic weight lifter Olympic victory Olympic

achievement

Olympic weight lifter Olympic victory Olympic

achievement

Description based on “salient” pixels only

Itti & Koch, 2001Bruce Tsotsos, 2009Judd et al., 2009Previous talk of the session

Liu et al, 2007

Grab-Cut, Rother et al., 2004

Our goal: Convey the image content

Stas GofermanLihi Zelnik-ManorAyellet Tal

1. Principles for context-aware saliency

2. A saliency detection algorithm

3. Applications:• Image retargeting

1. Principles for context-aware saliency

2. A saliency detection algorithm

3. Applications:• Image retargeting• Collages

Following perceptual properties

Local low-level factors• Contrast• Color• …

Walther & Koch, 2006

Global considerations• Maintain unique features

Hou & Zhang, 2007

Local & global• Should be multi-scale

Liu et al, 2007

Visual organization (Gestalt)• Few centers of gravity• Position is important!!

Our foci

High-level• Faces• Objects• People• …

Judd et al, 2009

Low-level

With face detection

Our result

Our result

LocalWalther & Koch, 2006

GlobalHou & Zhang, 2007

Local + globalLiu et al, 2007

The steps of our algorithm

Principles 1-2:Unique appearance salient

salient

Not salient

Principles 1-2:Unique appearance salient

Principles 1-2:Unique appearance salient

( , )color i jd p p Euclidean distance between colors of patches at pi & pj

Principles 1-2:Unique appearance salient

high ( , )color i jd p p j

salient

Principle 3:Position is important!

Similar patches both near and far

Not salient

Principle 3:Position is important!

Similar patches near

Salient

Principle 3:Position is important!

( , )position i jd p p Normalized Euclidean distance between positions of pi & pj

Distance between a pair of patches:

( , )( , )

1 ( , )color i j

i jposition i j

d p pd p p

c d p p

Distance between a pair of patches:

( , )( , )

1 ( , )color i j

i jposition i j

d p pd p p

c d p p

salient

High j

Distance between a pair of patches:

( , )( , )

1 ( , )color i j

i jposition i j

d p pd p p

c d p p

salient

High for K most similar

k

rq K most similar patches at scale r

1

11 exp ( , )

i j

Kr r ri

k

S d p qK

1

11 exp ( , )

i j

Kr r ri

k

S d p qK

Salient at:• Multiple scales foreground• Few scales background

1

1 Mrr

i ir r

S SM

Scale 1 Scale 4

Principle 3:• Few centers of gravity

Context

Foci =

Include distance map

0.8iS

1 ( )focid i

X

iS

Realizing Principles 1,2,3 at multiple scales

ˆ 1 ( )i i fociS S d i

Principle 4:• Faces• Objects• …

Excluded from this talk

Single-scale saliency

Multiple scales

Final saliency

1

11 exp ( , )

i j

Kr r ri

k

S d p qK

ˆ 1 ( )i i fociS S d i

1

1 Mrr

i ir r

S SM

X

…

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Database of Hou & Zhang

Liu et al, 2007

Our result

Image retargetingCollage

Seam Carving Our result

Seam Carving Our result

Seam Carving Our result

New definition: Context-aware saliency

Algorithm:Based on 4 perceptual principles

Applications

salient

Not salient

…