A Hybrid Region Weighting Approach A Hybrid Region Weighting Approach for Relevance Feedback in Region-for Relevance Feedback in Region-

Based Image Search on the WebBased Image Search on the Web

Deok-Hwan Kim, Jae-Won Song, Ju-Hong Lee

INHA University

23年 4月 22日

오전 7시 7분2

Contents

• Problem of CBIR

• Proposed RBIR

• Hybrid Region Weighting

• Experiment and Results

• Conclusion

23年 4月 22日

오전 7시 7분3

Content Based Image Retrieval

• CBIR– utilizes unique features (shape, color, texture) of

images

Users prefer– To retrieve relevant image by semantic

categories

– But, CBIR can not capture high-level semantics in user’s mind

23年 4月 22日

오전 7시 7분4

Problem of CBIR (1)

• Problem– Focused on developing effective global features

Can not capture properties of an object

The gap between low-level feature and high-level semantics

Query Image

User

System

Semantic Gap

23年 4月 22日

오전 7시 7분5

Problem of CBIR(2)

• Solutions– Relevance Feedback (RF)

– Region-based Image Retrieval (RBIR)

23年 4月 22日

오전 7시 7분6

Relevance Feedback

• Relevance Feedback– Learns the associations between high-level

semantics and low-level features

• Relevance Feedback Phase1. User identifies relevant images within the

returned set

2. System utilizes user feedback in the next roundTo modify the query (to retrieve better results)

3. This process repeats until user is satisfied

23年 4月 22日

오전 7시 7분7

Problem of CBIR(2)

• Solutions– Relevance Feedback (RF)

– Region-based Image Retrieval (RBIR)

23年 4月 22日

오전 7시 7분8

RBIR(Region-Based Image Retrieval)• Region-Based approaches

– Represent image at the object level

• The main objective – Enhance the ability of capturing user’s perception– More meaningful retrieval

• Image similarity measures– EMD

• Weighting of region – key factor of similarity definition.

23年 4月 22日

오전 7시 7분9

Proposed RBIR Approach

Q =(q,d,w, k)

Q =(q,d,w, k)

Image

Segmentation

Feature extraction

Image DB

Top k Retrieved Set

Region basedCluster Set

ClusterRepresentatives

EMD match

WeightComputation

Relevant Set

User Feedback Loop

Adaptive Clustering

Region 1

Feature extraction

WeightComputation

Region n

23年 4月 22日

오전 7시 7분10

Adaptive Region Clustering(1)

• Merges similar regions in the relevant set reduce retrieval speed

– T2 > Threshold : separate two clusters

– T2 <= Threshold : merge two clusters

)()( 12jipooledji

ji

ji xxSxxnn

nnT

))'()()'()((1

11jjk

n

kjjkiik

n

kiik

jipooled xxxxxxxx

nnS

ji

)(1,1

)2(

pnnppnn

nnp

jiji

ji FThreshold

23年 4月 22日

오전 7시 7분11

Adaptive Region Clustering(2)

•

Region

Image

C1 C2 Cm Cn

Ck Cg

C1

C1

Cg-1

gth level

(g-1)th level

23年 4月 22日

오전 7시 7분12

Image Segmentation and Region Representation

• Normalized cut segmentation– Discriminate foreground object regions and

background regions

• Region Representation in an Image– Twelve dimensional color and shape features– Color feature

• mean, standard deviation of color in L*a*b color space

– Shape feature• Compactness and convexity, region size, region location, and

variance of region pixels from the region center of mass

23年 4月 22日

오전 7시 7분13

Region Weighting

• Existing Region Weighting – Area Percentage

– RF (Region Frequency) * IIF (Inverse Image Frequency)

• Suggested Region Weighting– spatial locations of regions

– region size in an image

23年 4月 22日

오전 7시 7분14

Hybrid Region Weighting

• Assume that more important region– appear in center area of an image– tend to occupy larger area

• To consider image’s Spatial location

1

1

2)( 11

ra

xf

aexf )(2

raxf )1()(3

l

cxa

2

S

l

RA

x1

x2x3

xn

random pointsCenter C

RB

RC

)( 1Axf

)( 2Axf

)( nAxf

23年 4月 22日

오전 7시 7분15

Hybrid Region Weighting

• Region Importance – Calculated by summarizing the reciprocal function values with r

espect to all pixel locations x of region Rki

• However, it is difficult– Instead, use the asymptotic distance function by applying the M

onte-Carlo method

ik

i Rk dxxfRI )()(

,1

)()()(1

,,

i

i

ik

iii

kk

n

jjkjkk n

RxInxfRI

for 0

for 1 ) (

,

,

,

ii

ii

iikjk

kjk

kjk Rx

RxRxIn

rn

j k

k

jk

ik

i

i I

Rxf

1

1, 1)(

23年 4月 22日

오전 7시 7분16

Hybrid Region Weighting

• Region Weight

• Decay Factor β (0≤ β ≤1) reduce the effect of previous relevant image

• We assume that there are n relevant images I1…In

– prior images : I1…Im

– new images : Im+1… In

,)(

)()(

1

k

i

i

i n

j k

kksp

RI

RIRW 1)(

1 i

k

ksp

n

jRW

23年 4月 22日

오전 7시 7분17

Hybrid Region Weighting

• New region weights using decay factor is as follows:

kkspksp nimkRwRwii

1,1),1)(()(

kkspksp ninkmRwRwii

1,1),()(

,)(

)()(

1

v

j CR ksp

CR ksp

l

jik j

lik i

Rw

RwCCRI

v

j jCCRI1

1)(

23年 4月 22日

오전 7시 7분18

Three weights for regions of an animal image

area percentage 0.47region frequency 0.39area & location 0.15

area percentage 0.16region frequency 0.10 area & location 0.06

area percentage 0.06region frequency 0.23area & location 0.18

area percentage 0.01region frequency 0.11area & location 0.07

area percentage 0.06region frequency 0.12area & location 0.12

area percentage 0.24region frequency 0.04area & location 0.42

23年 4月 22日

오전 7시 7분19

Experiment and Results(1)

• k-NN query – used to accomplish the similarity-based match– k = 100.

• For RBIR with RF approach– Use adaptive region clustering method

• 10,000 general purpose color images from COREL • 40 random initial query • five feedback • For decay factor, empirically, β =0.3• To evaluate the performance, we compare

– Area percentage, Region frequency, Area & location

23年 4月 22日

오전 7시 7분20

Experiment and Results(2)

Performance evaluation

23年 4月 22日

오전 7시 7분21

Conclusion

• The main contribution– Calculate the importance of regions by using the hybrid

weighting method

– Cumulate it based on user’s feedback information to better represent semantic importance of a region in a given query

– Proposed weighting method can also be incorporated into any RBIR system on Web

– It put more emphasis on the latest relevant images that express the user’s query concept more precisely

• Experimental results– Show the superiority of the proposed method over other

weighting methods in terms of efficiency and effectiveness