prompt: personalized user tag recommendation for social...
TRANSCRIPT
PROMPT: Personalized User Tag Recommendation for Social Media PhotosLeveraging Personal and Social Contexts
Rajiv Ratn ShahNational University of Singapore
Anupam SamantaIndian Institute of Technology Dhanbad
Deepak GuptaIndian Institute of Technology Dhanbad
Yi YuNational Institute of Informatics
Suhua TangThe University of Electro-Communications
Roger ZimmermannNational University of Singapore
Abstract—Social media platforms such as Flickr allow usersto annotate photos with descriptive keywords, called, tags withthe goal of making multimedia content easily understandable,searchable, and discoverable. However, manual annotation isvery time-consuming and cumbersome for most users, whichmakes it difficult to search relevant photos. Moreover, predictedtags for a photo are not necessarily relevant to users’ interests.Thus, it necessitates for an automatic tag prediction system thatconsiders users’ interests and describes objective aspects of thephoto such as visual content and activities. To this end, thispaper presents a tag recommendation system, called, PROMPT,that recommends personalized tags for a given photo leveragingmultimodal information. Specifically, first, we determine agroup of users who have similar tagging behavior as the user ofthe photo, which is very useful in recommending personalizedtags. Next, we find candidate tags from visual content, textualmetadata, and tags of neighboring photos, and recommendsfive most suitable tags. We initialize scores of the candidatetags using asymmetric tag co-occurrence probabilities andnormalized scores of tags after neighbor voting, and laterperform random walk to promote the tags that have many closeneighbors and weaken isolated tags. Finally, we recommendtop five user tags to the given photo. Experimental results ona Flickr dataset (46,700 photos in the test set and 28 millionphotos in the train set) with 1,540 unique user tags confirmthat the proposed algorithm outperforms state-of-the-arts.
Keywords-Tag recommendation; personalized user tags; so-cial media; multimodal information; multimodal analysis
I. INTRODUCTION
PROMPT stands for a personalized user tag recommen-dation for social media photos leveraging multimodal in-formation. It leverages knowledge structures from multiplemodalities such as the visual content, textual metadata,user details, and semantically similar neighbors of a givenphoto to predict personalized user tags. Since the numberof photos on social media platforms has increased rapidly(e.g., Flickr has over ten billion photos) due to advance-ments in smartphone and digital camera technologies, itrequires an automatic tag recommendation system for an
efficient multimedia search and retrieval. User tags are veryhelpful in providing several significant multimedia-relatedapplications such as a landmark recognition [7] and a tag-based photo search and group recommendation [9]. A fewautomatic photo annotation systems based on visual conceptrecognition algorithms are proposed [2], [8]. However, theyhave limited performance because classes (tags) used in thetraining of deep neural networks to predict tags for a photoare restricted and defined by a few researchers and not byactual users. Thus, it necessitates a tag recommendationsystem that exploits tagging behaviors of other similar users.
The PROMPT system enables people to automaticallygenerate user tags for a given photo leveraging knowledgestructures from visual content, textual metadata, spatialinformation, and semantically similar neighboring photos.Moreover, it exploits information from past photos annotatedby a user to understand the tagging behavior of the user,which is useful in recommending personalized user tags. Inthis study, we consider the 1,540 most frequent user tagsfrom the YFCC100M dataset, a collection of 100 millionmedia records from Flickr (see Section IV for details), forthe tag prediction task. We construct a 1,540-dimensionalfeature vector, called, the UTB vector, to represent a user’stagging behavior using the bag-of-words model (see Sec-tion III for details). We cluster users and their photos inthe train set with 28 million photos into several groupsbased on cosine similarities among UTB vectors duringpre-processing. Moreover, we construct a 1,540-dimensionalfeature vector for a given photo, called, the photo description(PD) vector, using the bag-of-the-words model, to computethe photo’s N nearest semantically similar neighbors. UTBand PD vectors help PROMPT to find an appropriate setof candidate photos and tags for the given photo. SincePROMPT focuses on candidate photos instead of all photosin the train set for tag prediction, it is relatively fast. Weadopt the following approaches for tag recommendation.
Test
photooo
UTB vector
User groupsU
...PD vector
Tag co-occurrence
matrix
Tag frequncy matrix
Tag co-occurrence
matrix
Tag frequncy matrix
Group-1G 1
Group center vector
Tag co-occurrence
matrix
Tag frequncy matrix
Tag co-occurrence
matrix
Tag frequncy matrix
Group-MG M
Group center vector
Tag1
Tag2
Tag3
Tag4
Tag5
Predicted
tags
Candidate sets
Photo1
Tag1
Tag2
…
TagT
PhotoP Tag1
...
Neighbor
Voting
Based
Approach
Random
Walk
Based
Approach
P
F
u
s
i
o
n
Tag Co-
occurrence
Based
Approach
Test
set
Train
set
Figure 1. System overview of the PROMPT system.
• Often a photo consists of several objects and it is de-scribed by several semantically related concurrent tags(e.g., beach and sea) [24]. Thus, our first approach isinspired by employing asymmetric tag co-occurrencesin learning tag relevance for a given photo.
• Many times users describe similar objects in theirphotos using the same descriptive keywords (tags) [9].Hence, our second approach for tag recommendation isinspired by employing neighbor voting schemes.
• Random walk is frequently performed to promote tagsthat have many close neighbors and weaken isolatedtags [10]. Therefore, our third approach is based onperforming random walk on candidate tags.
• Finally, we fuse knowledge structures derived fromdifferent approaches to recommend the top five per-sonalized user tags for the given photo.
In the first approach, the PROMPT system first determinesseed tags from visual tags (see Section IV for more detailson visual tags) and textual metadata (excluding user tags)such as the title and description of a given photo. Next, wecompute top five semantically related tags with the highestasymmetric co-occurrence scores for all seed tags, and addthem to the candidate set of the given photo. Next, wecombine all seed tags and their co-occurred tags in thecandidate set using a sum method (i.e., if some tags appearmore than once then their relevance scores are accumulated).Finally, the top five tags with the highest relevance scoresare predicted for the given photo. In the second approach,our tag recommendation system first determines the closestuser group for the user of the given photo based on the user’spast annotated photos. Next, it computes the N semanticallysimilar nearest neighbors for the given photo based on thePD vector constructed from textual metadata (excluding usertags) and visual tags. Finally, we accumulate tags from allsuch neighbors and compute their relevance scores based ontheir vote counts and prior frequency in the train set, andrecommend the top five tags to the given photo.
In our third approach, we perform a random walk oncandidate tags derived from visual tags and textual metadata.The random walk helps in updating scores of candidate tagsiteratively leveraging exemplar and concurrent similarities.
Next, we recommend the top five user tags when the randomwalk converge. Finally, we investigate the effect of fusion bycombining candidate tags derived from different approachesand next perform a random walk to recommend the topfive user tags to the given photo. Experimental results on atest set of 46,700 Flickr photos (see Section IV for details)confirm that our proposed approaches outperform state-of-the-arts in terms of precision, recall, and accuracy scores.Steps of recommending user tags by PROMPT for a socialmedia photo is summarized as follows (see Figure 1).
• It determines a group of users from the train set with259,149 unique users, having similar tagging behavioras the user of the photo, based on cosine similarity.
• The candidate sets of photos and tags for the photo arecomputed from the selected user group.
• Relevance scores are computed for candidate tags usingour proposed approaches. Finally, the top five user tagsare recommended to the photo.
The paper is organized as follows. In Section II, we reviewrelated work and Section III describes the PROMPT system.The evaluation results are presented in Section IV. Finally,we conclude the paper with a summary in Section V.
II. RELATED WORK
Our purpose is to automatically predict personalized usertags for a given photo. The steps of such a process isdescribed as follows: (i) learn personal interests of a userfrom photos uploaded in past by the user, (ii) determine agroup of users who have similar tagging behavior as the user,and (iii) recommend tags for the photo leveraging knowledgestructures derived from its personal and social contexts. Inthis section, we briefly provide some recent progress oncomputing tag relevance for social media photos.
Learning tag relevance for a photo based on accumu-lating votes from visually similar neighboring photos is apopular approach [9], [18]. Research works in multimediaanalytics suggest that multimodal information is very usefulin several significant social media related applications andservices [15], [16]. For instance, multimodal information isvery useful in semantics and sentics understanding of user-generated content [3], [12], [20], [25]. Shah et al. [17],[19] leveraged visual tags and textual metadata (e.g., title,description, and user tags) to build semantics and senticsengines. Sigurbjornsson and Van Zwol [21] presented a tagrecommendation system to predict tags based on tag co-occurrence for each user input tag and merge them into asingle candidate list using their proposed aggregate (Vote orSum) and promote (descriptive, stability and rank promo-tion) methods. Anderson et al. [1] presented a tag predictionsystem for Flickr photos which combines both linguistic andvisual features of a photo. Further, Rae et al. [13] proposedan extendable framework that can recommend additionaltags to partially annotated photos using a combination ofdifferent personalised and collective contexts such as (i) all
Seed
Tags
...
Co-occurrence
Scores
...
Relevance
Scores
Predicted
Tags
S d
Photo
v1:s1
v2:s2
vn:sn
...
v3:s3
P di t d
Photo
v1:s1
v2:s2
vn:sn
v2:s2
vn:sn
...
t11:r11t11:r11...
t15:r15
tn1:rn1tn1:rn1...
PPhh tt
tn5:rn5
r11:s1 . p11
...r12:s1 . p12
r15:s1 . p15
t11:p11
t12:p12
t15:p15
...
tn1:pn1
tn2:pn2
tn5:pn5
...
rn1:sn . pn1
...rn2:sn . pn2
rn5:sn . pn5
...
Co-occurrence
Scores
...
Relevance
Scores
r11:s1 . p11
...r12:s1 . p12
r15:s1 . p15
t11:p11
t 12
t15:p15
t12:p12
t15:p15
...
Co occcurrence
tn1:ppn1
t n2
tn5:ppn5
tn2:ppn2
tn5:pn5
...
RRele ance
rn1:sn . ppn1
...rn2:sn . ppn2
rn5:sn . ppn5
Additional Tags With High Asymmetric
Co-occurrence Probabilities With Seed Tags
Figure 2. System framework of the tag prediction system based onasymmetric co-occurrence scores.
photos in the system, (ii) a user’s own photos, (iii) photos ofthe user’s social contacts, and (iv) photos posted in groups ofwhich the user is a member. Garg and Weber [4] proposeda system that suggests related tags to user, based on thetags that she or other people have used in the past alongwith (some of) the tags already entered. The suggestedtags are dynamically updated with every additional tagentered/selected. However, these approaches are not fullyautomatic since they expect a user to input (annotate) a fewinitial tags.
Wu et al. [24] formulated tag recommendation as a learn-ing problem and proposed a multimodal recommendationsystem based on both tag and visual correlation. Eachmodality is used to generate a ranking feature and Rankboostalgorithm is applied to learn an optimal combination of theseranking features from different modalities. Liu et al. [10]proposed a tag ranking scheme, aiming to automatically ranktags for a given photo according to their relevance to thephoto content. They estimated initial relevance scores of tagsbased on probability density estimation, and then performeda random walk over a tag similarity graph to refine relevancescores. Wang et al. [23] proposed a novel co-clusteringframework, which takes the advantage of networking infor-mation between users and tags in social media, to discoverthese overlapping communities. They clustered edges insteadof nodes to determine overlapping clusters (i.e., a singleuser belongs to multiple social groups). Such social groupsare also useful in determining aesthetic tendencies andcommunities [5], [6], [26]. Recent works [11], [14] exploituser context for photo tag recommendation.
III. SYSTEM OVERVIEW
Figure 1 shows the system framework of the PROMPTsystem. We compute user tagging behavior (UTB) vectorsfor all users based on their past annotated photos usingthe bag-of-the-words model on a set of 1,540 user tags T ,used in this study. We exploit UTB vectors to perform thegrouping of users in the train set and compute asymmetrictag co-occurrence scores among all 1,540 user tags for each
group during pre-processing. Moreover, the center of a groupis computed by averaging UTB vectors of all users in thatgroup. Similarly, we compute photo description (PD) vectorsfor photos using the bag-of-the-words model on tags inT . We do not consider user tags of photos to constructtheir PD vectors, instead we leverage tags derived fromtitle, description, and visual tags which belong to T . PDvectors are used to determine semantically similar neighborsfor photos based on the cosine similarity metric. Duringonline processing to recommend user tags for a photo, wefirst compute its UTB vector, and subsequently a closestmatching user group from the train set. We refer to theset of photos and tags in the selected user group as thecandidate set and further use them in recommending usertags for photos using the following techniques.
A. Asymmetric Co-occurrence based Relevance Scores
As described in literature [24], tag relevance is stan-dardized into mainly asymmetric and symmetric structures.Symmetric tag co-occurrence tends to measure how similartwo tags are, i.e., high symmetric tag co-occurrence scorebetween two tags indicates that they most likely to occurtogether. However, asymmetric tag co-occurrence suggestsrelative tag co-occurrence p(ti/tj), i.e., it is interpreted asthe probability of a photo being annotated with tj given thatit is already annotated with ti. Thus, an asymmetric tag co-occurrence score is beneficial in introducing diversity to tagprediction, and it is defined as follows.
pij = p(ti, tj) =|ti ∩ tj ||ti|
(1)
where |ti| and |ti ∩ tj | represents the number of times thetag ti appears alone and with tag tj , respectively.
Figure 2 describes the system framework of the tag pre-diction system based on asymmetric co-occurrence scores.We first determine seed tags from the textual metadata andvisual tags of a given photo. Seed tags are the tags appearedin title, visual tags, and description of the photo, whichbelong to the set of 1,540 user tags used in this study. Weadd seed tags and their five most co-occurred tags with thehighest asymmetric tag co-occurrence probabilities to thecandidate set of the photo. For all visual tags of the photo,their confidence scores si are also given as the part of theYFCC100M dataset. We set confidence scores of seed tagsfrom textual metadata as 1.0. For additional co-occurred tagsin the candidate set, we compute their relevance scores rijor r(ti, tj) as follows:
rij = r(ti, tj) = pij × si (2)
where, si is the confidence score of a seed tag and pij is theasymmetric tag co-occurrence probability of the tag tj forthe given seed tag ti. This formula is justifiable because itassigns the high relevance score to tj when the confidenceof the seed tag ti is high. In this way, we compute relevance
Predicted tags
Ground truth:
cat, kitty, kitten,
film, expired
cat
best …. pet kitty kitten film tabby animal feline cat
...
film, cat,
feline,
tabby
cat
cat,
whiskers,
yawn, best,
animal, pet
wash, cat,
garden, tabby,
home, animal,
pet, outdoor
cat, street,
grass, kitten,
eyes, brown
locomotora,
...,
train,
outdoor
Figure 3. System framework of the tag recommendation system based onneighbor voting scores.
scores of all tags in the candidate set. Next, we aggregate alltags and merge scores of common tags. Finally, we predicttop five tags with the highest relevance scores from thecandidate set to the photo.
B. Neighbor Voting based Relevance Scores
Earlier works [9], [18] on computing tag relevance forphotos confirm that a neighbor voting based approach is veryuseful in determining tag ranking. Leveraging personal andsocial contexts, we apply this approach for tag recommen-dation. Relevance scores of tags for a photo is computed inthe following two steps. Firstly, N nearest neighbors of thephoto are obtained from the user group of similar taggingbehaviors. Next, the relevance score of a tag t for the photois obtained as follows:
z(t) = vote(t)− prior(t,N) (3)
where z(t) is the tag t’s final relevance score, vote(t)represents the number of votes tag t gets from the Nnearest neighbors of the photo. prior(t,N) indicates theprior frequency of the tag t and is defined as follow:
prior(t,N) = NMt
D(4)
where Mt is the number of photos tagged with t, and D isthe size of the train set.
C. Random Walk based Relevance Scores
Another very popular technique for tag ranking is basedon random walk. Liu et al. [10] estimates initial relevancescores for tags based on probability density estimation, andthen perform a random walk over a tag similarity graphto refine the relevance scores. We leverage the multimodalinformation of photos and apply this tag ranking approachfor tag recommendation. Specifically, first, we determinecandidate tags leveraging multimodal information such asthe textual metadata (e.g., title and description) and thevisual content (e.g., visual tags). We estimate the initialrelevance scores of candidate tags adopting a probabilistic
approach on co-occurrence of tags. We also use the normal-ized scores of tags derived from neighbor voting. Next, werefine relevance scores of tags by implementing a randomwalk process over a tag graph which is constructed bycombining an exemplar-based approach and a concurrence-based approach to estimate the relationship among tags. Theexemplar similarity φe is defined as follows:
φe = exp(− 1
N ∗N∑
x∈Γti,y∈Γtj
||x− y||2
σ2) (5)
where Γt denotes the representative photo collection of tag tand N is the number of nearest neighbors. Moreover, σ is theradius parameter for the classical Kernel Density Estimation(KDE) [10]. Next, the concurrence similarity φc betweentag ti and tag tj is defined as follows:
φc = exp(−d(ti, tj)) (6)
where the distance d(ti, tj) between two tags ti and tj isdefined as follows.
d(ti, tj) =max(logf(ti), logf(tj))− logf(ti, tj)
logG−min(logf(ti), logf(tj))(7)
where f(ti), f(tj), and f(ti, tj) are the numbers of photoscontaining tags ti, tj , and both ti and tj , respectively, inthe training dataset. Moreover, G is the number of photos inthe training dataset. Finally, the exemplar similarity φe andconcurrence similarity φc are combined as follows:
Φij = λ · φe + (1− λ) · φe (8)
where λ belongs to [0,1]. We set it to 0.5 in our study.We use uk(i) to denote the relevance score of node i at
iteration k in a tag graph with n nodes. Thus, relevancescores of all nodes in the graph at iteration k form acolumn vector uk ≡ [uk(i)]n×1. An element qij of this n×ntransition matrix indicates the probability of the transitionfrom node i to node j and it is computed as follows:
qij =Φij∑k Φik
(9)
The random walk process promotes tags that have manyclose neighbors and weakens isolated tags. This process isformulated as follows.
uk(j) = α∑i
uk−1(i)qij + (1− α)wj (10)
where wj is the initial score of a tag tj and α is a weightparameter between (0, 1).
D. Fusion of Different Tag Recommendation Approaches
The final recommended tags for a given photo is deter-mined by fusing different above-mentioned approaches. Wecombine candidate tags determined by asymmetric tag co-occurrence and neighbor voting schemes. Next, we initializescores of the fused candidate tags with their normalized
Photo Details:
User Id, Visual Tags,
Description, Title, etc.
User
Groups
Candidate
Tags
Relevance
Score
Exemplar
Similarity
Concu-
rrence
Similarity
Iterations
Converge
Tag1
Tag2
Tag3
Tag4
Tag5
Predicted
Tags
User
Groups
Candidate
Tags
Figure 4. Architecture of the tag prediction system based on random walk.
scores from [0,1]. Further, we perform a random walkon a tag graph which has the fused candidate tags as itsnodes. This tag graph is constructed by combining exem-plar and concurrence similarities and useful in estimatingthe relationship among the tags. In this way, the randomwalk refines relevance scores of the fused candidate tagsiteratively. Finally, our PROMPT system recommends thetop five tags with the highest relevance scores to the photo,when the random walk converges.
IV. EVALUATION
A. Dataset
We used the YFCC100M [22] dataset consisting of 100million media records (approximately 99.2 million photosand 0.8 million videos) from Flickr. The reason for selectingthis dataset is its volume, modalities, and metadata. Forinstance, each media of the dataset consists of severalmetadata annotations such as user tags, spatial and temporalinformation, and others. Moreover, all media are labelledwith automatically added (visual) tags with confidencescores derived from a convolution network, which indicatethe presence of a variety of concepts such as ocean, food,and scenery, say, with confidence 95%, 85%, and 92%,respectively. There are a totally 1,756 visual tags present inthe YFCC100M dataset. The YFCC100M dataset has beensplit into 10 parts based the last digit prior to the @-symbolin their Flickr user identifier (NSID). Such split ensuresthat no user occurs in multiple partitions, thus avoidingdependencies between the different splits. Split 0 is usedas the test set and the remaining nine splits as the train set.
For tag prediction, a specific subset of 1,540 user tagsT are considered since predicting the correct tags from avirtually endless pool of possible tags is extremely chal-lenging. Tags in T fulfill the following criteria: (i) theyare valid English dictionary words, (ii) such tags do notrefer to persons, dates, times or places, (iii) they appearfrequently with photos in the train and test sets, and (iv)different tenses/plurals (tags) of the same word (an alreadyadded tag in T ) are not considered. The train set containsall photos from the YFCC100M that have at least one tagthat appeared in T and do not belong to the split 0. Thereare approximately 28 million photos present in the train set.The test set contains 46,700 photos from the split 0 suchthat each photo has at least five tags from the list of 1,540
Comparison K = 1 K = 3 K = 5
Accuracy@K Type-1 0.410 0.662 0.746Type-2 0.422 0.678 0.763
Precision@K Type-1 0.410 0.315 0.251Type-2 0.422 0.326 0.262
Recall@K Type-1 0.062 0.142 0.188Type-2 0.064 0.147 0.197
Table IRESULTS FOR THE TOP K PREDICTED TAGS.
tags. There are totally 259,149 and 7,083 unique users inthe train and test sets for this study, respectively.
B. Results
Recommended tags for a given photo in the test setare evaluated based on the following three metrics: (i)Precision@K, i.e., proportion of the top K predicted tagsthat appear in user tags of the photo, (ii) Recall@K, i.e.,proportion of the user tags that appear in the top K predictedtags, and (iii) Accuracy@K, i.e., 1 if at least one of the topK predicted tags is present in the user tags, 0 otherwise.PROMPT is tested for the following values of K: 1, 3,and 5. We implemented two baselines and proposed a fewapproaches to recommend personalized user tags for socialmedia photos. In Baseline1, we predict the top five mostfrequent tags from the training set of 28 million photos toa test photo. Further, in Baseline2, we predict five visualtags with the highest confidence scores (already providedwith the YFCC100M dataset) to a test photo. Since state-of-the-arts for tag prediction [1], [21] mostly recommendtags for photos based on input seed tags. In our PROMPTsystem, first, we construct a list of candidate tags usingasymmetric co-occurrence, neighbor voting, probability den-sity estimation techniques. Next, we compute tag relevancefor photos through co-occurrence, neighbor voting, randomwalk based approaches. We further investigate the fusion ofthese approaches for tag recommendation.
Figures 5, 6, and 7 depicts scores@K for accuracy,precision, and recall, respectively, for different baselines andapproaches. For all metrics, Baseline1 (i.e., recommendingthe five most frequent user tags) performs worst and thecombination of all three approaches (i.e., co-occurrence,neighbor voting, and random walk based tag recommen-dation) outperforms rest. Moreover, the performance ofBaseline2 (i.e., recommending the five most confident visualtags) is second from last since it only considers the visualcontent of a photo for tag recommendation. Intuitively,accuracy@K and recall@K increase for all approaches whenwe the number of recommended tags increases from 1 to 5.Moreover, precison@K decreases for all approaches whenwe increase the number of recommend tags. Our PROMPTsystem recommends user tags with 76% accuracy, 26%precision, and 20% recall for five predicted tags on the testset with 46,700 photos from Flickr.
Table I depicts accuracy, precision, and recall scores when
0.033 0.110.170.197
0.4320.517
0.307
0.515
0.625
0.401
0.5990.67
0.393
0.6140.701
0.418
0.6150.687
0.395
0.6160.701
0.419
0.6130.686
0.41
0.6620.746
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
K = 1 K = 3 K = 5
Baseline1 Baseline2 Co-occurrence (CO) Random-Walk (RW) CO+RW Voting Voting+CO Voting+RW Voting+CO+RW
Figure 5. Accuracy@K, i.e., user tag prediction accuracy for K predicted tags for different approaches.
0.033 0.038 0.037
0.197 0.180.148
0.307
0.2290.195
0.401
0.277
0.213
0.393
0.287
0.229
0.418
0.296
0.236
0.395
0.292
0.239
0.419
0.295
0.235
0.41
0.315
0.251
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
K = 1 K = 3 K = 5
Baseline1 Baseline2 Co-occurrence (CO) Random-Walk (RW) CO+RW Voting Voting+CO Voting+RW Voting+CO+RW
Figure 6. Precision@K, i.e., the precision of tag recommendation for K recommended tags for different approaches.
0.005 0.017 0.0270.03
0.081
0.11
0.046
0.102
0.145
0.061
0.125
0.16
0.06
0.13
0.172
0.063
0.133
0.176
0.059
0.131
0.178
0.063
0.133
0.175
0.062
0.142
0.188
0
0.05
0.1
0.15
0.2
0.25
K = 1 K = 3 K = 5
Baseline1 Baseline2 Co-occurrence (CO) Random-Walk (RW) CO+RW Voting Voting+CO Voting+RW Voting+CO+RW
Figure 7. Recall@K, i.e., recall scores for K predicted tags for different approaches.
a combination of co-occurrence, voting, and random walkis used for tag prediction. Type-1 considers a comparisonas a hit if a predicted tag matches ground truth tags andType-2 considers a comparison as a hit if either a predictedtag or its synonyms match ground truth tags. Intuitively,accuracy, precision, and recall scores are slightly improvedwhen the Type-2 comparison is made. This is consistent withall baselines and approaches which we used in our studyfor tag prediction. All results reported in Figures 5, 6, and7 correspond to the Type-1 match. Finally, Figure 8 showsthe ground truth user tags and the tags recommended by oursystem for five sample photos in the test set.
V. CONCLUSIONS
The proposed PROMPT system is an automatic tag rec-ommendation system. First, it determines a group of userswho have similar tagging behavior as the user of a givenphoto. Next, we construct lists of candidate tags for differentapproaches based on co-occurrence and neighbor voting.
Further, we compute relevance scores of candidate tags.Next, we perform a random walk process on a tag graph withcandidate tags as its nodes. Relevance scores of candidatetags are used as initial scores for nodes and updated in everyiteration based on exemplar and concurrent tag similarities.The random walk process iterates until it converges. Finally,we recommend the top five tags with the highest scores whenthe random walk process terminates. Experimental resultsconfirm that our proposed approaches outperform baselinesin personalized user tag recommendation. These approachescould be further enhanced to improve accuracy, precision,and recall in the future.
ACKNOWLEDGMENTS
This research was supported in part by Singapores Min-istry of Education (MOE) Academic Research Fund Tier1, grant number T1 251RES1415, and by JSPS KAKENHIGrant Number 16K16058.
User tag list (ground truth):
tent, black, beach, hotel, resort,
tourist, holiday
Predicted tag list:
beach, sea, coast, shore, nature
User tag list (ground truth):
cat, kitty, kitten, film, expired
Predicted tag list:
cat, film, kitty, pet, tabby
User tag list (ground truth):
surf, ocean, coast, sunset, foam,
silhouette
Predicted tag list:
beach, coast, water, ocean, sunset
Figure 8. Demonstration of recommended and ground truth user tags.
REFERENCES
[1] A. Anderson, K. Ranghunathan, and A. Vogel. TagEz: FlickrTag Recommendation. In AAAI Conference on ArtificialIntelligence, 2008.
[2] K. Barnard, P. Duygulu, D. Forsyth, N. d. Freitas, D. M. Blei,and M. I. Jordan. Matching Words and Pictures. In Journalof Machine Learning Research, 3(Feb):1107–1135, 2003.
[3] E. Cambria, S. Poria, A. Gelbukh, and K. Kwok. A Common-sense based API for Concept-level Sentiment Analysis. InWorkshop on Making Sense of Microposts at WWW, pages27–32, 2014.
[4] N. Garg and I. Weber. Personalized, interactive tag recom-mendation for flickr. In ACM Conference on RecommenderSystems, pages 67–74, 2008.
[5] R. Hong, L. Zhang, and D. Tao. Unified Photo Enhancementby Discovering Aesthetic Communities from Flickr. In IEEETransactions on Image Processing, 25(3):1124–1135, 2016.
[6] R. Hong, L. Zhang, C. Zhang, and R. Zimmermann. FlickrCircles: Aesthetic Tendency Discovery by Multi-view Regu-larized Topic Modeling. 18(8):1555–1567, 2016.
[7] L. Kennedy, M. Naaman, S. Ahern, R. Nair, and T. Ratten-bury. How Flickr Helps us Make Sense of the World: Contextand Content in Community-contributed Media Collections. InACM Multimedia, pages 631–640, 2007.
[8] J. Li and J. Z. Wang. Real-time Computerized Annotationof Pictures. In IEEE Transactions on Pattern Analysis andMachine Intelligence, 30(6):985–1002, 2008.
[9] X. Li, C. G. Snoek, and M. Worring. Learning SocialTag Relevance by Neighbor Voting. IEEE Transactions onMultimedia, 11(7):1310–1322, 2009.
[10] D. Liu, X.-S. Hua, L. Yang, M. Wang, and H.-J. Zhang. TagRanking. In ACM WWW, pages 351–360, 2009.
[11] A. O. Nwana and T. Chen. Who Ordered This?: ExploitingImplicit User Tag Order Preferences for Personalized ImageTagging. In arXiv preprint arXiv:1601.06439, 2016.
[12] S. Poria, E. Cambria, and A. Gelbukh. Deep ConvolutionalNeural Network Textual Features and Multiple Kernel Learn-ing for Utterance-level Multimodal Sentiment Analysis. InEmpirical Methods on Natural Language Processing, pages2539–2544, 2015.
[13] A. Rae, B. Sigurbjornsson, and R. van Zwol. ImprovingTag Recommendation using Social Networks. In Adaptivity,Personalization and Fusion of Heterogeneous Information,pages 92–99, 2010.
[14] Y. Rawat and M. S. Kankanhalli. ConTagNet: ExploitingUser Context for Image Tag Recommendation. In ACMMultimedia, 2016.
[15] R. R. Shah. Multimodal Analysis of User-Generated Contentin Support of Social Media Applications. In ACM Int’lConference on Multimedia Retrieval, pages 423–426, 2016.
[16] R. R. Shah. Multimodal-based Multimedia Analysis, Re-trieval, and Services in Support of Social Media Applications.In ACM Multimedia, pages 1425–1429, 2016.
[17] R. R. Shah, A. D. Shaikh, Y. Yu, W. Geng, R. Zimmermann,and G. Wu. EventBuilder: Real-time Multimedia Event Sum-marization by Visualizing Social Media. In ACM Multimedia,pages 185–188, 2015.
[18] R. R. Shah, Y. Yu, S. Tang, S. Satoh, A. Verma, and R. Zim-mermann. Concept-Level Multimodal Ranking of FlickrPhoto Tags via Recall Based Weighting. In MMCommonsWorkshop at ACM Multimedia, 2016.
[19] R. R. Shah, Y. Yu, A. Verma, S. Tang, A. D. Shaikh, andR. Zimmermann. Leveraging Multimodal Information forEvent Summarization and Concept-level Sentiment Analysis.In Knowledge-Based Systems, 108(Sep):102–109, 2016.
[20] R. R. Shah, Y. Yu, and R. Zimmermann. ADVISOR: Person-alized Video Soundtrack Recommendation by Late Fusionwith Heuristic Rankings. In ACM Multimedia, pages 607–616, 2014.
[21] B. Sigurbjornsson and R. Van Zwol. Flickr Tag Recommen-dation based on Collective Knowledge. In ACM WWW, pages327–336, 2008.
[22] B. Thomee, B. Elizalde, D. A. Shamma, K. Ni, G. Friedland,D. Poland, D. Borth, and L.-J. Li. YFCC100M: The NewData in Multimedia Research. In the Communications of theACM, 59(2):64–73, 2016.
[23] X. Wang, L. Tang, H. Gao, and H. Liu. DiscoveringOverlapping Groups in Social Media. In IEEE InternationalConference on Data Mining, pages 569–578, 2010.
[24] L. Wu, L. Yang, N. Yu, and X.-S. Hua. Learning to Tag. InACM WWW, pages 361–370, 2009.
[25] Y. Yin, Z. Shen, L. Zhang, and R. Zimmermann. Spatial-temporal Tag Mining for Automatic Geospatial Video An-notation. ACM Transactions on Multimedia Computing,Communications, and Applications, 11(2):29, 2015.
[26] L. Zhang and R. Zimmermann. Flickr Circles: MiningSocially-aware Aesthetic Tendency. In IEEE InternationalConference on Multimedia and Expo, pages 1–6, 2015.