Selective Aggregated Descriptors for Robust MobileImage Retrieval

Jie Lin ∗, Zhe Wang†, Yitong Wang†, Vijay Chandrasekhar∗, Liyuan Li ∗

∗ Institute for Infocomm Research, A*STAR, SingaporeE-mail: lin-j,lyli,vijay @i2r.a-star.edu.sg

† Institute of Digital Media, the School of EE & CS, Peking University, Beijing, ChinaE-mail: zhew,wangyitong@pku.edu.cn

Abstract—Towards low latency query transmission via wirelesslink, methods have been proposed to extract compact visual de-scriptors on mobile device and then send these descriptors to theserver at low bit rates in recent mobile image retrieval systems.The drawback is that such on-device feature extraction demandsheavy computational cost and large memory space. An alternateapproach is to directly transmit low quality JPEG compressedquery images to the server, but the lossy compression resultsin compression artifacts, which subsequently degrade featurediscriminability and deteriorate the retrieval performan ce. In thispaper, we present selective aggregated descriptors to address thisproblem of mobile image retrieval on low quality query images.The proposed mechanism of selective aggregation largely reducesthe negative impact of noisy features caused by compressionartifacts, enabling both low latency query transmission frommobile device and effective image retrieval on the server end. Inaddition, the proposed method allows fast descriptor matchingand less storage of visual descriptors for large database. Extensiveexperiments on benchmark datasets have shown the consistentsuperior performances of the proposed approach over the state-of-the-art.

I. I NTRODUCTION

Smart phones and Tablet PCs have shown great potentialsin mobile image retrieval [3][20], thanks to the integratedfunctionality of high resolution color camera and broadbandwireless network connection. Many mobile image retrievalapplications (such as Google Goggles [1] and Amazon Flow[2]) have been developed for retrieving similar images con-taining a rigid object in a large set of database images, givena query image of that object (such as CD/book cover, poster,logo, landmark, etc). In general, most mobile image retrievalsystems follow the client-server architecture. A capturedqueryis sent through the wireless network to the server, where imageretrieval is conducted to identify the relevant images fromanimage database stored on the server.

To reduce delivery latency for better user experience, theupstream query data is expected to be as small as possible,especially for unstable or limited bandwidth wireless connec-tion. Recent works have proposed to extract compact visualdescriptors of query images on the mobile device, and thensend such descriptors over a wireless link at low bit rates (seeFig. 1(a)). The compact descriptors extraction [8][11][10][19]follows a typical pipeline: statistics of local invariant features(such as SIFT [12] and SURF [13]) are aggregated to forma fixed-length vector representation, which is subsequently

compressed into compact descriptors.Aside from low latency query delivery, such on-device

descriptors extraction may demand heavy computational costas well as memory footprint, making it impractical to workwith mobile devices that have limited processor power andRAM. One example is local feature detection and description(such as SIFT [12]) on mobile device typically takes a fewseconds. Another example is locally aggregated descriptors(such as Bag-of-Words (BoW) [15][22]) often involves a largevisual vocabulary containing 0.1-1 million visual words. Thevocabulary would cost hundreds of megabytes to be loaded inthe limited RAM of mobile device.

As smart phones are hardware friendly to support fast JPEGimage compression at extremely low cost, an alternate ap-proach is to directly transmit JPEG compressed query imagesover a wireless link, the subsequent descriptor extractionandmatching are performed on the server side (see Fig. 1(b)). Onecan reduce the compressed image size by decreasing the JPEGquality factor, however, the lossy compression introducescom-pression artifacts appeared in low quality images. It is noticedthat compression artifacts would degrade the discriminabilityof detected local features, and deteriorate the retrieval andmatching performance (see Section IV). As shown in Fig. 2,the number of inlier matches (i.e., true positive matches)is largely reduced with the quality of query image (from100 to 5). Thus, it is crucial to consider a mechanism thatincorporates the selection of informative local features intodescriptor extraction.

In this paper, we present selective aggregated compactdescriptors to address the problem of low quality imageretrieval. Our contributions are three fold. First, state-of-the-artlocally aggregated descriptors [9][8] unfairly assume that alllocal features extracted from an image contribute equally to thesubsequent aggregation stage, resulting in suboptimal retrievalaccuracy. We propose a selective aggregation to reduce thenegative impact of noisy local features on the aggregateddescriptors. Second, we propose to model the characteristicsof match/non-match keypoint pairs for selecting informativelocal features, with the observation that true positive matchkeypoints are statistically associated with informative localfeatures (see Fig. 2). Third, the proposed approach enablesboth low bit rate query transmission (e.g., 6 KB per image)and effective image retrieval. In addition, the proposed method

Compact

Descriptors

Extraction

Wireless

Network

g

JPEG

Compressed

Image

Descriptor

Matching

Compact

Descriptors

Extraction

Descriptor

Matching

Query Data

Query Data

Top Ranked

Results

(a)

(b)

Fig. 1. Low bit rate mobile image retrieval frameworks: (a)Extracting and compressing visual descriptors on the mobiledevice, and sending the compact codesover a wireless link (Top), and (b)Transmitting highly compressed JPEG query images, and subsequent descriptors extraction and matching are performed onthe server (Bottom).

Quality = 100, Image Size = 163kB

Query Image Reference Image

Quality = 5, Image Size = 5.45kB

Quality = 100

Image Size = 107kB

Fig. 2. Negative effect of lossy JPEG compression on local invariant featurematching. A highly compressed query image (Quality = 5) degrades thediscriminability of detected features, which subsequently reduces the numberof inlier matches (highlighted in green). The informative local featuresdetected by the proposed method are denoted in red, while thenoisy onesare in yellow.

allows fast descriptor matching as well as light storage of com-pact descriptors extracted from large scale database images.

Our extensive experiments on benchmark datasets (such asUKbench) combined with 1 million distractor images haveshown the consistent superior performances of the proposedapproach over the state-of-the-art [9][8][10]. For example,mean average precision (mAP) is improved from 33.9% to69.4% on Graphics dataset (JPEG qualityQ = 30 for queryimages), compared to the compressed Fisher vector [8].

II. RELATED WORK

Local invariant features like SIFT [12] and SURF [13]cannot meet the requirement of compactness, as the size oflocal descriptors usually exceeds the size of raw image itself.There are a lot works on compressing local descriptors [7][6].For instance, Chandrasekhar et al. [6] proposed a CompressedHistogram of Gradient (CHoG), which adopts Gagie treecoding to compress each local feature into approximate 60bits. Suppose that 1,000 keypoints are detected per image, theoverall feature size is approximately 8KB.

Recent work stepped forward to compute statistics of localfeatures and then aggregate them to form a fixed-length vectorrepresentation, which is subsequently compressed for efficientstorage and transmission. The Bag-of-Words (BoW) represen-tation [14][15][25] is the most widely adopted method for thispurpose. Each local feature from an image is quantized to itsclosest visual word in a visual vocabulary. BoW accumulatesthe number (0-order statistics) of local features assignedtoeach visual word. Chen et al. [21] and Ji. et al. [5][4] proposedto further compress the quantized BoW histogram.

Recently, the Fisher Vector (FV) [8] extends the BoW bycomputing higher-order statistics of the distribution of localfeatures, e.g., Gaussian Mixture Model (GMM). Specifically,FV aggregates the gradient vector of each local feature’slikelihood w.r.t. the GMM parameters (mean or variance) foreach Gaussian. Jegou et al. [9] proposed a non-probabilisticFV, namely, the Vector of Locally Aggregated Descriptors(VLAD), to aggregate residual vectors (difference betweenlocal feature and its nearest visual word). Both FV and VALDcan be compressed into compact binary codes [8][9] for fastHamming distance computation. Chen et al. [10] introducedthe Residual Enhanced Visual Vector (REVV), where lin-ear discriminant analysis (LDA) is employed to reduce thedimensionality of VLAD, followed by sign binarization togenerate compact codes. Lin et al. [19] further improvedthe compactness of these binary aggregated descriptors byprogressively coding informative sub-vectors, which achieves

comparable retrieval accuracy to the raw FV or VLAD.

III. A GGREGATING INFORMATIVE LOCAL FEATURES

In this section, we introduce the selective aggregated com-pact descriptors. We first give an overview of the proposedmethod in III-A. In III-B, we formulate the selective aggrega-tion as a keypoint ranking problem, and implement the key-point ranking by employing likelihood ratio test with GaussianMixture Models (GMM) that fit the empirical distribution ofmatch/non-match keypoint pairs. Finally, we discuss how toestimate the parameters of the GMM distribution in III-C.

A. Overview

We illustrate the pipeline of locally aggregated compactdescriptors in Fig. 3. Three stages including feature coding,aggregation and compression, are usually adopted to generatea compact descriptor.

Feature coding. Let I = (zt,xt)Tt=1 denote a collection

of d-dimensional local featuresx and their detected keypointsz in image I. In this work, we focus on the SIFT feature(d = 128), and the keypointz = (η, θ, v, ξ) is of fourdimensions, whereη, θ, v and ξ denote scale, orientation,peak value in scale space and the distance from a keypoint tothe image center, respectively. The goal of feature coding isto embed local featuresx in a visual vocabulary space basedon a encoderr:

r : x ∈ Rd → r(x) ∈ Rd. (1)

Specifically, the BoW approach obtains a codebookQ by k-means clustering, whereQ = q1, ...,qK is comprising ofKvisual words, and the encoderr quantizes each local featureto its nearest visual wordq1NN from Q:

r(x)BoW = q1NN . (2)

The VLAD encodes each local feature to its residual error:

r(x)V LAD = x− q1NN . (3)

The FV [16] extends the discrete k-means clustering to prob-ability GMM clustering. We denote the GMM codebook as:qk = ωk, µk, σ

2k, k = 1, ...,K, whereωk, µk andσ2

k are theweight, mean vector and variance vector of thekth Gaussian(visual word), respectively. In this work, we derive the codesas the gradient of local feature’s likelihood w.r.t. the mean µk

of each Gaussian:

r(x)FV = γ(k)σ−1k (x− µk), (4)

whereγ(k) = ωkpk(x)/∑K

l=1 ωlpl(x) denotes the probabilityof local featurex being assigned to thekth Gaussian.

Feature aggregationaccumulates the feature codes of localdescriptors into a fixed-length vector representation for animage. State-of-the-art approaches usually employ averagepooling to aggregate the feature codes for each visual word:

g(k) =∑

x∈Xk

f(r(x)), (5)

whereXk represents the subset of local features in imageI thatare assigned to thekth visual word.f(·) denotes an operationon the feature codesr(x). In the case of BoW,f(·) simplyrefers to the occurrences of each visual word in an image:

g(k)BoW =∑

x∈Xk

1. (6)

Thus, the dimensionality of BoW representation isK.While the VLAD and FV directly accumulate the residual

vectors:g(k)V LAD =

∑

x∈Xk

r(x)V LAD, (7)

g(k)FV =∑

x∈Xk

r(x)FV . (8)

Finally, the VLAD and FVg are formed by concatenating thesub-vectorsg = [g(0), ..., g(K)] of all visual words and isthereforeKd-dimensional.

Feature compressionaims to compress high dimensionalaggregated descriptorsg into binary codes, which supportsultra-fast Hamming distance computation (XOR operation andbit count) as well as light storage of features for large scaleimage retrieval. For instance, the Compressed Fisher vector(CFV) [8] proposed to quantize each dimension of the FVrepresentation into a single bit based on a sign function.Formally, we project each elementg of descriptorsg to 1if g > 0; otherwise, 0:

sgn(g) =

1 if g > 00 otherwise.

(9)

Problem definition From Eq. 5, we observe that existingapproaches unfairly assume that all local features contributeequally to the aggregation stage. As shown in Section IV,it significantly degenerates the retrieval accuracy, especiallyfor lower quality JPEG queries. To address the problem, wepropose a selective aggregation that injects a weight termw(z)associated with local featuresx into Eq. 5:

g(k) =∑

x∈Xk

w(z)f(r(x)), (10)

wherew(z) is defined over the detected keypointsz. In thiswork, we model the termw(z) as a keypoint ranking functionbased on likelihood ratio test, which determines whether thecorresponding local featuresx are involved in aggregation ornot.

B. Keypoint Ranking for Selective Aggregation

From the matching point of view (see Fig. 2), informativelocal features tend to be associated with true positive matchkeypoints between images. Thus, to fulfill the selective ag-gregation of local features, we propose to learnw(z) fromthe perspective of patch-level keypoint matching. In particular,we employ likelihood ratio test to accomplish the selectionoflocal features for subsequent aggregation [18]:

w(z) =p(z|H0)

p(z|H1), (11)

Coding

Local

featuresCompact

descriptorAggregation Compression

1101010011…101

Fig. 3. The extraction pipeline of locally aggregated compact descriptors.

where hypothesisH0 and H1 represent whether keypointzwould be a correct match or not, respectively.p(z|Hi), i = 0, 1is the probability density function for hypothesisHi, alsoreferred to as the likelihood of the hypothesisHi given akeypoint samplez. The likelihood functions can be learnedfrom match and non-match keypoint pairs, using the key-points’ characteristicsz (e.g., scale, orientation, etc). Duringa test, we compute the values ofp(z|H0) and p(z|H1) fora given keypointz, and predict the likelihood ratio ofzbeing correctly matched or not using Eq. 11. We note thatw(z) → ∞ if p(z|H0) → 1 and p(z|H1) → 0, which is therequired objective.

In statistics, the hypothesis test has been widely appliedto test if a certain statistical model fits the samples [23].In this wok, we first train a universal modelp(z|λH1

) withparametersλH1

for hypothesisH1 over the entire keypoint setBH1

detected from training images. Rather than independentlylearning the modelp(z|λH0

) for hypothesisH0, we adoptBayesian adaptation to deriveλH0

smoothly by updating thewell-trained parametersλH1

of the universal model usingthe match keypoint setBH0

generated from match imagepairs. Bayesian adaptation is a popular modeling approach inspeech and speaker recognition [23], which is able to harvestsufficient prior knowledge about the distribution of keypointsvia the universal model.

Once the weightw(z) is computed for each keypoint de-tected from an image, we propose to rankw(z) of the detectedkeypoints in descending order, and then produce a binarydecision for each local featurex. Specifically, if w(z) is inthe topτ highest weight values, the corresponding descriptorx is adopted in aggregation (i.e.,w(z) = 1); otherwise, it isdiscarded (i.e.,w(z) = 0).

C. Parameter Estimation

Constructing the training keypoint setBH1andBH0

. LetΩ = 〈Iln, I

rn〉

Nn=1 denoteN match image pairs,(Ze

n,Xen) =

(zenm,xenm)|e ∈ l, r,m = 1...Mn denote a collection

of detected keypointszenm and the corresponding descriptorsxenm extracted from imageIen. The entire keypoint setBH1

=zt|t = 1...B1, zt ∈ Ze

n.

We employ a distance ratio test [12] to compute match key-point pairsDn = 〈xl

nd,xrnd〉|d = 1...Dn from 〈Xl

n,Xrn〉,

which may remove many false matches from backgroundclutter. Subsequently, a geometric consistency check likeRANSAC [22] is applied to divideDn into inliers Dn =〈Xl

n, Xrn〉 = 〈xl

nd, xrnd〉|d = 1...Dn and outliersDn \ Dn.

The inliersDn are finally considered as true positive matches.Finally, we construct the match keypoint setBH0

= zt|t =1...B0, zt ∈ Ze

n, whereZen denotes the keypoints associated

with Xen. Note thatBH1

contains both match and non-matchkeypoints, whileBH0

is a subset ofBH1.

Estimating model p(z|λH1). Given the training setBH1

,we adopt a GMM model to learn the distribution of keypointfeaturesz as:

p(z|λH1) =

C∑

c=1

ωcpc(z), (12)

where λH1= ωc, µc, σ

2c

Cc=1, C denotes the number of

Gaussian components. The covariance matrices are assumedto be diagonal and the variance vector is denoted asσ2

c . Welearn the parametersλH1

by maximizing the likelihood ofBH1

.Estimating model p(z|λH0

). Given the match keypoint setBH0

and the universal modelp(z|λH1), we perform Bayesian

adaptation in twin-stage iteration. The first step is identical tothe expectation step of EM algorithm, which usesB0 keypointsamplesz from BH0

to calculate the sufficient statistics aboutthe GMM parameters of weight, mean and variance:

nc =

B0∑

b=1

γb(c), (13)

Ec(z) =1

nc

B0∑

t=1

γ(c)z, (14)

Ec(z2) =

1

nc

B0∑

t=1

γ(c)z2, (15)

where γ(c) = ωcpc(z)∑Cc=1

ωcpc(z)denotes the soft assignment of

keypointz to Gaussianc.

The second step is to apply the above sufficient statisticsfrom BH0

to update the parametersωc, µc, σ2c. The adapted

parametersλH0= ωc, µc, σ

2c

Cc=1 is derived as follows:

ωc = αwc nc/B0 + (1− αw

c )ωc, (16)

µc = αscEc(z) + (1− αs

c)µc, (17)

σ2c = αt

cEc(z2) + (1− αt

c)(σ2c + µ2

c)− µ2c , (18)

whereαwc , αs

c and αtc are adaptation coefficients to control

the impact of universal model on parameters updating. Forexample, whenαs

c is large, the statisticsEc(z) from matchedkeypoints tend to dominate in Eq. 17. In this work, we definethe coefficientsαρ

c , ρ ∈ w, s, t as the ratiosαρc = nc

nc+πρ ,whereπρ is a constant relevance factor for parameterρ andnc is defined in Eq. 13.

IV. EXPERIMENTAL RESULTS

A. Datasets and evaluation metrics

To evaluate the performance of the proposed approach, wecarry out retrieval experiments over public available datasets,including heterogeneous categories of objects and scenes (seeTable 1).

TheGraphics dataset depicts 5 product categories includingCDs, DVDs, books, text documents and business cards. Thereare 1,500 queries and 1,000 reference images. The queriesare captured by mobile phones under widely varying lightingconditions with foreground or background clutter.

ThePainting dataset contains 400 queries and 100 referenceimages for museum paintings, including history, portraits,landscapes and modern-art.

The Frame dataset is an image set of 500 video frames,containing diverse content like movies, news reports andsports. There are 400 queries taken by mobile phone fromlaptop, computer and TV screens to include typical speculardistortions.

TheLandmark dataset consists of 3,499 queries and 9,599reference images collected from landmarks and buildings fromthe world.

The UKbench dataset contains images of 2,550 objects.Each one has 4 images taken from different viewpoints.

Query image compression. Each query image is com-pressed by JPEG compression with quality factorQ =5, 10, 15, 20, 30, 50, 100. The compression artifacts becomestronger as the image quality decreases. Color images areconverted to gray ones to save bits, as local features areextracted from luminance component. For each compressionfactor, 8,349 query images are generated.

For large scale experiments. We use a datasetFLICKR1Mcontaining 1 million distractor images randomly downloadedfrom Flickr website. This image set is merged with thereference images to evaluate the accuracy and efficiency overa large-scale.

Evaluation measures. For all experiments we use the meanAverage Precision (mAP) to measure the search accuracy.

TABLE ISTATISTICS OF THE IMAGE DATASETS IN THE EXPERIMENTS.

Dataset # query images # database imagesGraphics 1,500 1,000Painting 400 100Frame 400 100Landmark 3,499 9,599UKbench 2,550 7,650FLICKR1M – 1,000,000In Total 8,349 1,018,449

Graphics

Landmark

UKbench

Frame

Painting

Fig. 4. Sample images from different test datasets.

mAP is defined as follows:

mAP =1

Nq

Nq∑

i=1

(

∑Nr=1 P (r)

# relevant images). (19)

whereNq is the number of queries;N the number of relevantimages for theith query; P (r) is the precision at rank r.We deploy both the SIFT feature extraction (see Fig. 1(a))and JPEG compression scheme (see Fig. 1(b)) on a HTCDESIRE G7 smart phone. This application is used to evaluatethe extraction time and memory cost on mobile client, as wellas the transmission delay of JPEG compressed query imagesover a WLAN link.

B. Experiment setup

All the images are resized with reduced resolutions (maxside≤ 640 pixels). SIFT features are extracted by the VLFeatlibrary. We employ independent image sets for all the trainingstages. Specifically, the MIRFLICKR25000 dataset is used totrain all the vocabularies (i.e. GMM and k-means codebook).To obtain training keypoint setBH1

and BH0, we use the

match/non-match image pairs from Oxford building and Cal-tech building datasets.

C. Baselines

(1) Bag-of-Words (BoW) [15]: We adopt hierarchical k-means clustering to train a vocabulary tree with depth 5 andbranch factor 10, resulting in a105 visual words. Inverted

index file is build up to implement efficient search. (2)Resid-ual Enhanced Visual Vector (REVV) [10]: Chen et al. applieddimensionality reduction and sign binarization to compress theVLAD representation. (3)Compressed Fisher vector (CFV)[8]: The work in [8] employed sign binarization to quantizeraw FV signature vector, which outperformed the state-of-the-art, such as Locality Sensitive Hashing and Spectral Hashing.(4) Product Quantized SIFT (PQ-SIFT) [17]: PQ-SIFT isthe state-of-the-art compact local descriptor [17], followingthe pipeline in Fig. 1(a) which extracts and quantizes rawSIFT features by a product quantization technique [24]. (5)Selective Aggregated BoW (BoW SA): the proposed selectiveaggregation scheme combined with BoW. (6)Selective Aggre-gated REVV (REVV SA): the proposed selective aggregationscheme combined with REVV. (7)Selective Aggregated CFV(CFV SA): the proposed selective aggregation scheme com-bined with CFV.

D. Impact of parameter τ

We first study the impact of the numberτ of aggregatedlocal features for CFV aggregation (QualityQ = 15). Asshown in Fig 5, the retrieval performance in terms of mAPover all test datasets is consistently improved whenτ increasedfrom 100 to 300, and the mAP rapidly decreases as the numberof selected features increased from 300 to CFVAll (i.e.,standard CFV). The optimalτ for all test datasets is about300. For instance, on the Graphics dataset,τ = 300 yieldsthe best mAP 65.8%. To make clear the advantage of theselective aggregation scheme, we produce the results of CFVaggregation by randomly sampling 300 SIFT descriptors fromeach query (see Fig 5). We can see that the mAP of CFVRandis even much worse than standard CFV, e.g. 24.3%vs. 42.4%on Landmark dataset. This demonstrates the power of selectiveaggregation as well.

E. Compression factor analysis

Fig. 6 shows the retrieval mAP vs. JPEG compressionfactors over different datasets. Firstly, the selective aggregationsignificantly outperforms the state-of-the-art over all datasetsat all compression factors. ForQ = 20, BoW, REVV andCFV yield mAP 40.1%, 24.1% and 36.0% on average over alldatasets, while the BoWSA, REVV SA and CFV SA haveachieved better mAP 59.1%, 62.8% and 64.2%, respectively.This gain may be attributed to the fact that the SA schemediscards less informative local features. Secondly, as thequal-ity factor Q increases, all methods improve the performanceprogressively. For example, mAP is improved from 65.4%with Q = 20 to 72.7% withQ = 50 for CFV SA on Paintingdataset . More importantly, we observe that the SA scheme atlow quality (e.g.,Q = 20) performs better than the baselines athigh quality (e.g.,Q = 100). For instance, on Frame dataset,BoW SA with Q = 20 outperforms BoW withQ = 100(mAP 72.8% vs. 63.8%) by∼10 times query size reduction(∼16KB vs. ∼170KB). The results demonstrated that theselective aggregation provides a trade-off between query sizeand search accuracy. Fig. 7 shows several groups of visualized

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

mAP

Number of SIFT

Graphics

Painting

Frame

Landmark

UKbench

Fig. 5. Influence of the number of selected SIFTs for CFV aggregation ondifferent datasets, combined with the 1 million distractorset FLICKR1M(Quality Q = 15 for query images).

TABLE IICOMPARISON OF THE PROPOSEDCFV SA (Q = 20) AND THE

STATE-OF-THE-ART PQ-SIFT,IN TERMS OF MAP (%) ON DIFFERENTDATASETS, COMBINED WITH THE 1 MILLION DISTRACTOR SET

FLICKR1M.

Dataset CFV SA PQ-SIFTGraphics 67.4 62.6Painting 65.4 72.8Frame 78.8 69.6

Landmark 51.3 42.8UKbench 58.2 37.8

retrieval performances using CFVSA with comparisons toCFV with Q = 50.

F. Comparison with the state-of-the-art

Table 2 compares the performance of CFVSA (Q = 20)with PQ-SIFT [17][24] for comparable query size transmissionon mobile client (i.e.,∼16KB). The CFV SA obtains bettermAP than PQ-SIFT over all datasets except the Paintingdataset. For instance, CFVSA achieves a much better mAP58.2% on UKbench, while PQ-SIFT reports 37.8%. This isprobably due to (1) PQ-SIFT causes considerable quantizationerror of local features and degenerates the subsequent retrievalperformance; (2) the selective aggregation is able to reduce thenegative impact of lossy JPEG compression.

TABLE IIIMEMORY AND COMPUTATION TIME COMPARISON BETWEENJPEG

COMPRESSION ANDSIFT EXTRACTION ON A HTC SMART PHONE BY

AVERAGING COSTS FROM1000 VGASIZE QUERY IMAGES.

Complexity JPEG compression SIFTMemory ∼0 ∼18MB

Computation time (s) ∼0.1 ∼2.2

Fig. 6. mAP vs. query image with different JPEG quality factors over various types of datasets, combined with the 1 million distractor set FLICKR1M.

TABLE IVQUERY IMAGE SIZE VS. TRANSMISSION TIME (VIA WLAN) AT DIFFERENT

JPEGQUALITY FACTORS ON A HTC SMART PHONE BY AVERAGINGTRANSMISSION TIME FROM1000 VGASIZE QUERY IMAGES.

Quality Factor Image Size (KB) Upload Time (s)5 6.65 0.2710 9.96 0.4715 12.91 0.5020 15.53 0.5530 20.49 0.8050 28.31 0.97100 169.36 4.58

G. Complexity analysis

Memory and computation time. Table 3 compares thememory and computation (extraction) time between JPEGcompression and SIFT extraction on a smart phone. Theresults show that directly sending a highly compressed JPEGimage provides prominent advantages in terms of memoryand computation time, compared to extracting local featuresdirectly on the mobile phone.

Transmission time. Table 4 reports the delivery time ofJPEG compressed query images with different quality factorsover a WLAN link. The time required to send the highestquality query image (Q = 100) is several times longer thanthe lower quality ones, which would cost serious energyconsumption. Fortunately, the selective aggregation supportslow quality query transmission, and battery saving may beexpected.

V. CONCLUSION

We have proposed discriminative locally aggregated com-pact descriptors by informative local feature selection. Theselective aggregation is able to reduce the negative effectof compression artifacts that appear in low quality JPEGquery images, which enables low latency query delivery aswell as power saving on the mobile client. In addition, theselective aggregation scheme supports fast similarity matchingof descriptors based on Hamming distance and light storageof large scale database images. Our approach has shownpromising retrieval performance over extensive benchmarks.

Fig. 7. The retrieval performance of CFVSA in comparison to CFV [8] with JPEG query qualityQ = 50. Each line corresponds to a query with top 10dataset images returned. Left: CFV; Right: CFVSA. The selected local features are denoted in red in query image, the noisy local features are in yellow.Green boxes indicate relevant image. The query images are randomly chosen from Graphics, Painting, Frame, Landmark andUKbench datasets.

REFERENCES

[1] Google Goggles. www.google.com/mobile/goggles.[2] Amazon Flow. www.a9.com/whatwedo/mobile-technology/flow-powered-

by-amazon.[3] B. Girod, V. Chandrasekhar, D. Chen andet al.. Mobile Visual Search.

IEEE Signal Processing Magazine, 2011.[4] R. Ji, L.-Y. Duan, J. Chen andet al.. Location discriminative vocabulary

coding for mobile landmark search.IJCV, 2012.[5] R. Ji, L.-Y. Duan, J. Chen andet al.. Towards Low Bit Rate Mobile

Visual Search with Multiple-Channel Coding.ACM Multimedia, 2011.[6] V. Chandrasekhar, G. Takacs, D. Chen andet al.. Compressed Histogram

of Gradients: A Low-Bitrate Descriptor.IJCV, 2012.

[7] V. Chandrasekhar, G. Takacs, D. Chen andet al.. Transform coding ofimage feature descriptors.VCIP, 2009.

[8] F. Perronnin, Y. Liu, J. Sanchez andet al.. Large-Scale Image Retrievalwith Compressed Fisher Vectors.CVPR, 2010.

[9] H. Jegou, M. Douze, C. Schmid andet. al. Aggregating local descriptorsinto a compact image representation.CVPR, 2010.

[10] D. Chen, S. Tsai andet al. Residual enhanced visual vector as a compactsignature for mobile visual search.Signal Processing, 2012.

[11] H. Jegou, F. Perronnin, M. Douze andet. al. Aggregating local imagesdescriptors into compact codes.PAMI, 2012.

[12] D. G. Lowe. Distinctive image features from scale invariant keypoints.IJCV, 2004.

[13] H. Bay, T. Tuytelaars, and L. Van Gool. SURF: Speeded up robust

features.ECCV. 2006.[14] J. Sivic and A. Zisserman. Video Google: A Text Retrieval Approach

to Object Matching in Videos.ICCV, 2003.[15] D. Nister and H. Stewenius. Scalable recognition with avocabulary tree.

CVPR, 2006.[16] T. Jaakkola and D. Haussler. Exploiting generative models in discrimi-

native classifiers.NIPS, 1999.[17] J. Chen, L.-Y. Duan, J. Lin, andet al.. On the Interoperability of Local

Descriptors Compression.ICASSP, 2013.[18] J. Lin, L.-Y. Duan, T. Huang, and W. Gao. Robust Fisher Codes for

Large Scale Image Retrieval.ICASSP, 2013.[19] J. Lin, L.-Y. Duan, Y. Huang, andet al.. Rate-adaptive Compact Fisher

Codes for Mobile Visual Search.IEEE Signal Processing Letters, 2014.[20] L.-Y. Duan, J. Lin, J. Chen, andet al.. Compact descriptors for visual

search.IEEE Multimedia, 2014.[21] D. Chen , S. Tsai, V. Chandrasekhar andet al.. Tree Histogram Coding

for Mobile Image Matching.DCC, 2009.[22] J. Philbin, O. Chum, M. Isard, andet al.. Object Retrieval with Large

Vocabularies and Fast Spatial Matching.CVPR, 2007.[23] D. Reynolds, T. Quatieri, R. Dunn. Speaker VerificationUsing Adapted

Gaussian Mixture Models.Digital Signal Processing, 2000.[24] H. Jegou, M. Douze, C. Schmid. Product quantization fornearest

neighbor search.PAMI, 2011.[25] H. Jegou, M. Douze, C. Schmid. Improving Bag-of-Features for Large

Scale Image Search.IJCV, 2010.