Query Expansion for Hash-based Image Object Retrieval

Yin-Hsi Kuo, Kuan-Ting Chen,Chien-Hsing Chiang, Winston H. Hsu

National Taiwan UniversityTaipei, Taiwan

2

Our Goal–Effective and Efficient Object-level Image Retrieval

Ranking list

Object-level imageQuery image

- Object-level image retrieval

- Effective image retrieval

- Time efficiency

3

Problems in Conventional Content-Based Image Retrieval (CBIR)

• Global feature, e.g., color histogram– Not suitable for object-level image retrieval– Time-consuming for finding nearest neighbors

bin

Color histogram

bin

Color histogram

Query image Target image

4

Image (Many-to-many) Matching by Local Features

• Local feature, e.g., SIFT feature [D. Lowe. IJCV 2004]

– Measuring image similarities by local feature– Time-consuming for many-to-many matching– Requiring efficient indexing methods

• e.g., KD-tree, hashing

Query image Target image

5

• Hash-based indexing method (HB)

• Hash-based indexing + Query expansion (HQ)

5

Query image

Our Proposal–Query Expansion for Hash-based Image Retrieval

After query expansionPrecision-Recall curve

Recall

Precision

HB

HQ

Ideal case

Suffering from high precision but low recall

6

Two Strategies for Query Expansion

• Intra-expansion– Expanding more target feature points similar to query

• Inter-expansion– Mining feature points that shall co-occur with targets

• For example, query– New York– Musicals

- NYC, NewYork, New York City

- musical, show

Intra-expansion

- Cats- Mamma Mia- The Lion King- Directors- Times square

Inter-expansion

7

Highlights of Our Proposal

• Propose intra- and inter-expansion for hash-based image retrieval

• Object-level image retrieval• Hash-based method consumes <1 second to

retrieve images among 5000+ photos• Relative 76.3% (average) improvements over

the original hash-based method

8

Hash-based Indexing Methods

• E.g., Locality Sensitive Hashing (LSH)– Efficient similarity search of high-dimensional data– Similar feature points are mapped to the same

buckets with high probability

• Applicable for different hash functions– Stable distribution [M. Datar. SoCG 2004]

– Random projection [M. Charikar. STOCI 2002]

9

• Hash function: – v: feature point– a: stable distribution

• E.g., L2(Gaussian), L1(Cauchy)– b: 0 ~ W

Illustration of Locality sensitive hashing

⎥⎦⎥

⎢⎣⎢ +⋅

=W

bvavh )( v

Inner product

[M. Datar. SoCG 2004]

Hash table 1

1 2 3 4

3

2

1

Hash table 2

1

2

3

43

2

1High-dimensional feature space

Bucket

W

a

10

• Collecting all local features from images• Hashing them into buckets by LSH• Ranking result by score

LSH for Bags of Feature Points

Ranking scoreImageImageImage

H, I, J, K, L

A, B, C, D

E, F, G

Feature ID Query featureDatabase images Hash tables

[Y. Ke. ACM MM 2004]

11

Illustration of Query Expansion—Intra- and Inter-expansion

LSH

Target feature points:ABCD

B

B

B

A

A

C D

D

Ranking result

C

C

A

C

B

D

Query image

A’, A’’B’, B’’C’, C’’D’

Intra-expansion

B

B

B

A

A

C D

D

Intra-expansion

C

C

B’

B’’

A’’

A’C’

C’’

D’B’’

A’’

A’C’

C’’

D’

Inter-expansion

EFG

B

B

B

A

A

C D

D

Inter-expansion

C

C

E

G F

E

E

G

G

F

E

G F

E

E

G

G

F

A’AA’’

Hash table

12

Intra-expansion in Hashing

• Expanding more feature points similar to query• Possible solutions

– Matched points– Near buckets [Q. Lv. VLDB 2007]

– Meta-featureHash table 1

A C

DE G

F

H I J

B

R R

Hash table 2

C G

HA D

FE

I

J

B

13

Intra-expansion in Hashing—Matched Points

• Expanding more feature points from retrieved features within R radius distance– Red star → E, F, G

• E and G are matched points

– E → B (in hash table 2)

Hash table 1

A C

DE G

F

H I J

B

R R

Hash table 2

C G

HA D

FE

I

J

B

14

Intra-expansion in Hashing—Near Buckets

• Expanding more feature points from near buckets– Retrieve nearest bucket to the query feature– Then retrieve the second nearest bucket

Hash table 1

A C

DE G

F

H I J

B

R R

Hash table 2

C G

HA D

FE

I

J

B

[Q. Lv. VLDB 2007]

15

Intra-expansion in Hashing—Meta-feature

• Expanding more feature points base on the cluster center– Generate cluster centers to represent meta-features– Use meta-features to retrieve more possible

candidates

A C

DE G

F

H I J

B

Meta-feature

Hash table 1

A C

DE G

F

H I J

B

R

16

Inter-expansion in Hashing

• Mining feature points that shall co-occur with targets• Possible solutions

– Pseudo-relevance feedback (Top N)– Utilizing spatial verification to avoid false positive

images• Expanding feature

points from– Full image– Matched Region

Query image

Target image

Region B

Region A

17

Inter-expansion in Hashing—New Feature Points Retrieved from Target Images

Query image

Correct images(Inlier number ≧δ)

Ranking list 2B(Intra-

expansion) ……

Ranking list 1

(Intra-expansion)

A………

Ranking list 3

Inter-expansion

(Intra-expansion) ……

18

Fusion Methods for Inter-expansion Ranking Results

Ranking list 1

……

Ranking list 2

……

Ranking list 3

……

Final ranking list

…

• Average score• Maximum score• Borda count• Average inlier number• Maximum inlier number

Fusion

19

Experimental Dataset–Oxford Buildings Dataset

• Oxford buildings dataset– 5,062 images of particular Oxford landmarks

• 7M (7,162,122) feature points• 55 queries

– Object-based retrieval and landmark recognition

Ashmolean

Balliol

20

Experimental Dataset–Flickr11K Dataset

• Flickr11K dataset– 11,282 images downloaded from Flickr

• 10M (10,632,711) feature points• 56 queries

– More diverse and complicated

Torre PendenteDi Pisa

Starbucks logo

21

Experimental Result on Best Configuration

• Baseline– Hash-based indexing method

• Intra-expansion– Near buckets

• Inter-expansion– Matched region

• Intra- & inter-expansion– Using intra-expansion to retrieve initial ranking result– Doing inter-expansion base on initial ranking result

A C

DE G

F

H I J

B

R

Matched region

22

Experimental Result—Intra- and Inter-Expansion Complement Each Other

• MAP: mean average precision• %: relative improvement

• Combining both intra- and inter-expansion can achieve better results

OxfordMAP %

Baseline 0.261 -

75.00.2255.20.13567.30.215

-0.128%MAP

Flickr11K

52.00.396Intra-expansion

13.10.295Inter-expansion76.30.460Intra- & Inter-expansion

23

(a) Object-level query image

(f) Hash-based + Intra- and inter-expansion32 (3363) 33 (516) 34 (1948) 35 (445) 37 (727)

10 (45) 24 (88) 25 (116) 26 (641) 29 (67)(e) Hash-based + Inter-expansion

14 (779) 16 (22) 18 (72) 19 (37) 24 (548)(d) Hash-based + Intra-expansion

(c)(e) (d) (f)

Precision

Recall

1

0 1(b) PR curve

1 2 12 15 19(c) Hash-based ranking list

Oxford buildings

24

(c)

(e) (d) (f)

Precision

Recall

1

0 1(b) PR curve

1 2 4 6 7(c) Hash-based ranking list

4 (14) 7 (66) 8 (196) 9 (29) 11 (105)(e) Hash-based + Inter-expansion

5 (69) 8 (394) 9 (893) 11 (58) 12 (31)(d) Hash-based + Intra-expansion

(f) Hash-based + Intra- and inter-expansion8 (238) 22 (1387) 25 (183) 29 (1031) 33 (2065)

Flickr11K

(a) Object-level query image

25

Conclusions & Future Work

• Propose two novel expansion strategies–intra-expansion and inter-expansion

• Collaboratively boost the performance significantly in consumer photo benchmarks

• Optimize efficient implementations for the expansions and the spatial verification process

• Seek parallel solutions for further speeding up large-scale image object retrieval

Thank you for your attention!

Q & A