HeyLo: Visualizing User Interests from TwitterUsing Emoji in Mixed Reality

Hunter Harris, Makayla Thompson, Isaac Griffith, Paul BodilyDepartment of Computer Science

Idaho State UniversityPocatello, ID

harrhunt@isu.edu, thommaka@isu.edu, grifisaa@isu.edu, bodipaul@isu.edu

Abstract—We tackle the problem of analyzing a user’s interestsfrom social media content and subsequently visualizing theseinterests in an extended reality environment. We compare fivemodels for extracting interests from Twitter users and how wecan measure the effectiveness of these models. We also look at howthese interest extraction models fit in the context of HeyLo, anextended reality computational creativity (XRCC) framework forvisualizing potential conversational topics. The chosen interestsfor a particular person are visualized using emoji. We accomplishthis by using an emoji2vector model to find the closest relatedemoji to a given interest. We perform a comparative analysisbetween the five interest extraction models on real-world usersand their tweets, evaluating specificity, variance, and relevance.

Index Terms—extend reality, computational creativity, senti-ment analysis, twitter, emoji

Source code is available at https://github.com/harrhunt/HeyLo

I. INTRODUCTION

The growth of AI for improving interactions on social me-dia, online retail, and media-service recommendation systemssignals the importance placed on the ability of a systemto understand and leverage a user’s interests for effectivecommunication. Despite the value added from technologyin these domains, relatively unexplored is how to improvereal-life, day-to-day interactions using similar technologiesin mixed reality—a medium with the potential to directlyperceive and enhance a user’s experience with the world.Already technology has emerged to translate written languagein real time via augmented reality [1]. It is not difficultto imagine how similar systems may soon allow targetedadvertising on billboards or storefronts. These technologiescarry the potential to improve the effectiveness of all forms ofday-to-day interaction.

Interaction, and conversation in particular, represents animportant form of creativity. Effective dialogue hinges asmuch on generating novelty, value, surprise, and intention aseffective artefacts in any artistic domain. Much of what isessential in an effective conversation focuses around findingareas of common interest.

We have designed an extended reality computational creativ-ity (XRCC) framework called HeyLo. This system leveragesthe capabilities of mixed reality with the principles of compu-tational creativity in order to analyze user interests and visuallysuggest (using emoji) potential conversation topics based on

the areas of interest as determined from analyzing social me-dia. Our main interest in developing HeyLo is to research howXRCC can be used to enable a user to meaningfully engageand interact with another randomly encountered user. In thispaper, we focus on the design of the HeyLo system itselfand a comparative analysis of interest-extraction algorithmsfor use in HeyLo, leaving the analysis of human interactionusing HeyLo for future work.

Much of the design of HeyLo and the interest extraction andvisualization algorithms we compare rely on previous work inthese fields. Topic modeling, which is in some sense analogousto interest extraction, has been attempted using a variety ofmethods including Latent Dirichlet Allocation [2], [3], chi-square analysis [4], and online dynamic topic modeling [5].

Whereas both topic modeling and interest extraction rep-resent forms of semantic analysis, the latter is also morespecifically a form of sentiment analysis. Although severalsentiment analysis methods aim to classify text as merelypositive or negative (e.g., for categorizing product reviews),some models exist which seek to extract greater variety ofsentiment. The Empath [6] algorithm analyzes text acrosscustomizable lexical categories. The system requires the userto define a mapping from sets of words and phrases to eachcategory. Given some input text, Empath returns a set of key-value pairs where keys are lexical categories and values arescores for that category. This allows Empath to extract andrank topical and emotional content from a body of text.

Also critical are tools for identifying relationships betweenwords on the basis of sentiment. ConceptNet [7] is a semanticnetwork that helps computers understand the meaning humansput on words. It connects nodes represented as words andphrases with labeled edges that represent the relationshipbetween the words and phrases. These edges can be traversedusing the representational state transfer (REST) API.

Interest visualization is significantly aided by tools basedaround ideograms (e.g., emoji). Eisner et al. [8] trained anemoji2vector model (similar to a word2vec model but withemoji) using the Unicode emoji name and descriptions. Foreach word in the description, its vector representation retrievedfrom the Google News word2vec model is summed togetherto build a vector representation for the emoji.

Fig. 1. An overview of the HeyLo system. Given an image of a person (e.g., from an XR headset), the system recognizes the user, gathers tweets, extractsinterests, and outputs visualizations for display on the image. Picture of Bill Gates is in the public domain.

II. METHODS

In this section we describe the design and operation of theHeyLo XRCC system. We first provide a high-level overviewof the system. At the heart of this system is a model thatextracts user interests from a set of tweets. For purposesof comparison, we define five possible implementations forthis model. Finally we define four metrics by which wecomparatively evaluate the performance of each of these fiveimplementations.

A. HeyLo System Overview

Taking as input the image f of a person, the system outputsa user u and a set R of l weighted keywords representinginterests of user u (l = 5 for all of our examples). Each interestis paired with a representative emoji. The flow of informationthrough HeyLo is as follows. On input f :

1) Identify from the set of all users U (i.e., all availablesocial media users) the user u represented in f usingfacial recognition1.

2) Retrieve 1000 of the most recent social media postscreated by u and represent the content of these postsas a single multiset of words S.

1This step is envisioned as part of future work and is not currently employedas part of HeyLo.

3) Filter S for stopwords, URLs, and non-alphabetic char-acters.

4) For each word s ∈ S, replace s with the NLTK [9]WordNetLemmatizer lemmatized form of s.

5) Apply interest extraction model M on S to obtain a setof pairs IM,u = {(k1, w1), . . . , (ki, wi), . . . , (kn, wn)}where ki is a keyword or interest and wi ∈ R≥0represents a weight or level of interest for ki.

6) Reduce IM,u to the pairs (ki, wi) with the l highestweights wi.

7) For (ki, wi) ∈ IM,u select an emoji ei ∈ E (where E isthe set of all emoji) as follows:

a) Use word2vec to compute a vector representationvi for ki.

b) Identify an emoji ei from vi by computing

ei = argmine∈E

d(vi, ve)

where ve is the vector obtained by applyingword2vec to the text label for an emoji e (wherethe label is multiple words, ve is the average ofthe vectors for each word), and d is the functioncomputing the cosine distance of two vectors (i.e.,a measure of dissimilarity).

c) Add the triple (ki, wi, ei) to the return set R.

8) Output (u,R)

A system overview is depicted in Fig. 1.

B. Interest Extraction Models

Here we describe five possible implementations for modelM as described above in step 5. For each implementationwe define how the method generates a set of pairs IM,u ={(k1, w1), . . . , (kn, wn)} from a multiset of word lemmas S.

1) Empath: The Empath algorithm [6] uses a mappingfunction f : Σ∗ −→ C from the set Σ∗ of word strings toa set C of categories. Given a multiset of words S, Empathcomputes a score ω(cj) for each category cj ∈ C equal to thenumber of words si ∈ S such that f(si) = cj . Thus for theEmpath model,

IM,u = {(ki, wi)|ki ∈ C,wi = ω(ki)}.

In this implementation of M we use the pre-defined default setC (200 categories) and the default, human-validated mappingfunction f .

2) Retrained Empath: In a second implementation of M ,we retrained Empath on an expanded set of categories C ′ thatunions C with 1000 categories taken from Facebook’s listof page categories2. We automatically generated a mappingf ′ : S −→ C ′ (an augmentation of the function f described inthe previous section) by adding a mapping to each cj ∈ C ′

from words connected to cj via any (English) relationship inConceptNet [7]. Using C ′ and f ′, we retrained Empath togenerate a set of weighted interests as

IM,u = {(ki, wi)|ki ∈ C ′, wi = ω′(ki)},

where ω′(cj) for a category cj ∈ C ′ equals the number ofwords si ∈ S such that f ′(si) = cj .

3) Raw word counts: Given the multiset S, the raw wordcount model derives a keyword ki for each unique si ∈ S andderives a weight wi equal to the number of times ki appearsin S:

IM,u = {(ki, wi)|ki ∈ S,wi = |[s|s ∈ S, s = ki]|}.

4) Bayesian: Given the multiset S for user u, the Bayesianmodel derives a keyword ki for each unique si ∈ S and derivesa weight wi equal to

wi = Pr(u|ki) =Pr(ki|u)× Pr(u)

Pr(ki).

The distributions Pr(s|u), Pr(u), and Pr(s) were computedfrom the word counts for words s in the multisets S′ derivedfrom the last 1000 tweets from each of a set of 500 users U(ensuring that u ∈ U ). These users were selected from a listof the top 500 most followed Twitter handles3.

2https://www.facebook.com/pages/category/3https://socialblade.com/twitter/top/500/followers

5) Chi-square: The chi-square approach derives a keywordki for each unique si ∈ S and computes a weight wi equal tothe chi-square contribution of the occurrence of si for user u:

wi =(xi − xi)

2

xi

where xi is the expected occurrence of si for user u (ascomputed from U ), and xi is the actual number of times usaid si. As this method proved particularly effective, we showan example of the chi-square analysis in Table I.

C. Evaluation MetricsWe evaluated the output from each model M according to

four criteria which serve to define measures of quality andnovelty for the set of extracted interests: specificity, intra-uservariance, inter-user variance, and relevance. In defining theseterms we use IM,u to represent the set of interests extractedby M for a particular user u. For the purposes of evaluation,the weights of items in IM,u (used for the pruning in step 6from the system overview section) are ignored.

1) Specificity: In order to uniquely characterize the user’sinterests, a good model will return interests that are specificrather than general. For two words v and w, v IsA w indicatesthat v is more specific than w (e.g., “field lacrosse” IsA“sport”). We define in–degree(k) for a keyword k as thenumber of words v such that v IsA k is a valid relationshipcatalogued in ConceptNet. From this we define the specificityof an interest k as

specificity(k) = 1/(in–degree(k) + 1)

Note that if k has an in-degree of 0 (e.g., as with k = “fieldlacrosse”), k cannot be further categorized or specified. In thisscenario, k would receive the maximum specificity score of1.0.

Using the definition of specificity for a keyword, we definethe specificity of a model M . Let IM represent the set ofunique interests extracted by M across all users. Then thespecificity of M is the average of the specificity values foreach unique interest:

specificity(M) =

∑k∈IM

specificity(k)

|IM |.

2) Intra-user variance: In returning a set of interests repre-sentative of a user, a good model will extract a set of diverseinterests. For a set of interests IM,u extracted by model Mfor user u, we define the variance of IM,u as

varianceintra(IM,u) =∑

(ki,wi),(kj ,wj)∈IM,u

DC(vi, vj)

where DC(vi, vj) is the cosine distance between two vectorsvi and vj representing interests ki and kj . Using the definitionof intra-user variance for a set of interests, we define the intra-user variance of a model M as the average of the intra-uservariance values across all users:

varianceintra(M) =

∑u∈U

varianceintra(IM,u)

|U |.

@DaveRamsey @realDonaldTrump @tonyhawk @Nick Offerman @BillGates . . . Total

thank 12(96.62)

123(1.63)

90(2.51)

110(9.07)

17(83.15) . . . 33871

get 379(471.78)

158(0.32)

154(12.16)

106(2.68)

115(0.87) . . . 37070

impeachment 0(N/A)

140(4166.12)

0(N/A)

1(1.02)

0(N/A) . . . 1083

money 236(5560.14)

20(8.11)

11(0.93)

4(0.90)

18(9.38) . . . 2623

skate 0(N/A)

0(N/A)

160(30569.06)

0(N/A)

0(N/A) . . . 264

malaria 0(N/A)

0(N/A)

0(N/A)

0(N/A)

69(12311.65) . . . 113

sleep 3(0.46)

0(N/A)

1(2.20)

0(N/A)

1(2.49) . . . 1256

wood 0(N/A)

0(N/A)

1(0.66)

5(61.53)

1(0.54) . . . 144

time 120(7.49)

95(1.59)

82(0.02)

52(2.50)

94(0.20) . . . 26528

birdhouse 0(N/A)

0(N/A)

53(16162.62)

0(N/A)

0(N/A) . . . 55

......

......

......

. . ....

Total 11340 13106 10098 7846 10886 . . . 3216565

TABLE IWORD COUNTS (AND CHI-SQUARE CONTRIBUTIONS) FROM USER TWEETS

3) Inter-user variance: In addition to extracting a set ofdiverse interests, a good model will also extract diverse setsacross users or, in other words, avoid repeatedly extractingthe same interests for multiple users. To measure this inter-user variance, we find the sum of unique words for each userinterest set divided by the total unique words across all usersets:

varianceinter(M) =

|⋃

u∈UIM,u|∑

u∈U|IM,u|

.

4) Relevance: A model’s success depends on extractinginterests that are not only specific and varied, but which alsoreflect the user’s actual interests. This final metric may be themost important of all, but is also one of the most challengingaspects to measure. A model’s predictions for a user’s interestscan only be accurately assessed by the user him/herself. Weare planning to conduct such a study as future work.

III. RESULTS

To compare the performance of each of these methods,we preselected five Twitter handles for five widely-recognizedusers. We chose to perform the analysis on these users onthe basis that their interests are generally well-known andtherefore the results could be more easily compared to knownresults. A more thorough quantitative and qualitative analysisof these models using real users is envisioned as future work.

The top five interests for each model, together with interestweights, are shown in Table II. Weights are only useful for

comparing the level of interest for each keyword for a partic-ular user and model (i.e., the way in which weights are deriveddoes not allow for comparison across usernames or acrossmodels). Comparing the keywords and orderings for eachmodel against general knowledge about users’ interests formsthe basis for making arguments about a model’s relevance.

To objectively evaluate the specificity, intra-, and inter-user variance, we extracted interests using each model forover 500+ publicly available Twitter users. Results of thesecalculations are shown in Table III.

By the specificity and variance metrics the Bayesian modellooks to have performed the best. Looking at Dave Ramsey’sresults, for example, the Bayesian model extracts words thatare specific and varied (e.g., belay, godly, backache, variable,and toolbox). These results, however, have very low relevance.Knowing that Dave Ramsey is a businessman, author, and arenowned financial advisor, the interests extracted by the chi-square model are of significantly higher relevance (e.g., money,advice, financial, debt, and millionaire).

Both the default and retrained Empath models struggle sig-nificantly with inter-use variance, reflecting that these modelsextract many of the same words across several users. The lowspecificity score betrays that these model also extracts wordswhich are relatively non-descript (e.g., play and party). Theraw word count model also suffers from low specificity (e.g.,thank, thanks, and people). The Bayesian model has morespecific words for each user, however the nature of this modelleads it to prefer words that given the data are uniquely used

Interest Extraction ModelUsername Empath Retrained Empath Raw Word Count Bayesian Chi-square

@DaveRamsey

moneyeconomicsbusinessbankingvaluable

683638517515459

businesspartyweddingschoolwork

972728679556551

getmoneytodaythankslife

287236166129120

belaygodlybackachevariabletoolbox

1110.660.6

moneyadvicefinancialdebtmillionaire

5569.324630.434504.553642.452991.81

@realDonalTrump

negative emotioncommunicationspeakingachievementpositive emotion

459386365321298

healthcareleaderstandworkbusiness

655566538497491

presidenttrumpimpeachmentpeoplethank

321186140130123

legitimizeunfairnesscudgeladmissibilityfrack

11111

transcriptwitchimpeachmentdemocratshifty

5749.735618.874170.223759.483486.31

@tonyhawk

partyplaycelebrationmessagingsports

294284270251229

partystandplaynightwedding

609419418406401

thanksvideoseetodaythank

22399888686

staabprimusclubmanfamelesslasher

11111

skatebirdhousehawkdemolitionvert

30569.0616162.6213786.739016.615619.85

@Nick Offerman

positive emotioncommunicationpartyfriendsgiving

242198197196185

partyproducerstandhealthcarejoint

458364345336334

nickthankgetpleasesee

158108777667

woodworkerpawneeunsettlinggumptionapprobation

11111

nickmirthberrywoodworkerpawnee

21095.849908.164759.142039.821223.89

@BillGates

optimismpositive emotionreadingschoolscience

572473443442401

schoolweddingshowworkbusiness

1082871698682668

worldpeoplehelpprogressenergy

245196132112108

diagnosticshapeshifterthackerteletypepathogen

11111

malariapolioprogresspovertyvaccine

12311.657288.123269.682588.421970.48

TABLE IITOP FIVE INTERESTS (AND INTEREST SCORES) PER USER BY INTEREST EXTRACTION MODEL

Metric Empath Retrained Empath Raw Word Count Bayesian Chi-square

Average specificity 0.013± 0.020 0.020± 0.076 0.093± 0.238 0.677 ± 0.376 0.245 ± 0.350

Average intra-user variance 0.174 ± 0.069 0.141± 0.052 0.171 ± 0.072 0.113± 0.068 0.140± 0.081

Average inter-user variance 0.054 0.090 0.23 1.00 0.75TABLE III

COMPARATIVE ANALYSIS OF INTEREST EXTRACTION MODELS

by a particular user, regardless of their relevance. It standsto reason that as the amount of data in the model increasesthat the Bayesian model will improve, but in its current state,this model suffers from over-specificity. Like the results ofthe Bayesian model, those of the chi-square model exhibitkeywords that are varied and specific but without being toospecific.

The NULL emoji for Nick Offerman’s word pawnee (afictional city in a TV series featuring Offerman) was a resultof the word pawnee not being in the Google News word2vecmodel. The model is unable to find the closest associated emojibecause a vector for the word could not be determined. Futurework will seek to address this issue.

We concluded that as a combination of relevance, specificity,intra-, and inter-user variance, the chi-square model was theonly model that performed reasonably well across all metrics.The results of the full HeyLo interest visualization systemusing the chi-square model are depicted in Table IV.

IV. DISCUSSION AND CONCLUSION

The relevance of the keywords identified by the chi-squaremodel is noticeably higher than that of other models, in somecases to the point of being surprising. When the chi-squaremodel was applied to Bernie Sanders, one of the interests it

returned was insulin. Upon further investigation, we learnedthat Bernie Sanders is diabetic. This shows that the chi-squaremodel is able to find relevant interests to a given user that arenot obvious or trivial.

Because the models are defined to operate on text obtainedfrom social media, their ability to extract relevant interestsis limited to what users are open and willing to discuss onsocial media. This is a known limitation of our approach. Theapproach is not inherently limited to text from this medium,however, and as other suitable means for obtaining textualrepresentations of users interests (e.g., journals, private blogs,etc.) become available, our method is easily adapted to takethese sources as input.

Inter-user variance, while a helpful metric, can becomeproblematic if given too much emphasis. Where the goal isultimately to identify interests that users have in common.This problem should ideally be mitigated by finding waysto identify similar but non-identical interests or to proposeways to bridge dissimilar interests. In this case, inter-uservariance maintains its value as a metric for extracting varietyand specificity in users’ interests.

In this paper, we explored several different models forextracting a user’s interests from their tweets and visualizing

User Interests & Emoji Visualizations

@DaveRamsey

money

money-bag.png

advice

warning.png

financial

bank.png

debt

credit-card.png

millionaire

man-farmer.png

@realDonaldTrump

transcript

memo.png

witch

woman-fairy.png

impeachment

file-cabinet.png

democrat

man-office-worker.png

shifty

wavy-dash.png

@tonyhawk

skate

ice-skate.png

birdhouse

snowman.png

hawk

owl.png

demolition

building-construction.png

vert

skateboard.png

@Nick Offerman

nick

man-bouncing-ball.png

mirth

cat-with-tears-of-joy.png

berry

cherries.png

woodworker

man-artist.png

pawnee

NULL

@BillGates

malaria

mosquito.png

polio

cancer.png

progress

hourglass-not-done.png

poverty

children-crossing.png

vaccine

pill.png

TABLE IVTOP FIVE INTERESTS AND EMOJI VISUALIZATIONS FROM CHI-SQUARE INTEREST EXTRACTION MODEL

those interests as emoji. We discovered that the chi-squaremodel had the best results across the metrics we lookedat despite the Bayes model seeming to be a better fit. Wealso found the emoji2vector model described in [8] to be anexcellent way to visualize the interests as emoji.

Our system is limited to only being able to analyze textto discover a user’s interests. Our system also becomes lesseffective if the user does not write about their core interestsor has limited text in their social media. As seen in Table IV,our system can’t always provide a visualization for a giveninterest a user has.

Despite these limitations, our newfound tools will aid us indeveloping HeyLo as a system for used in social interactions.Our next steps are to find a way to compare interests betweenusers and bridge their seemingly dissimilar interests in a noveland surprising way. Our goal is to create an XRCC frameworkcapable of enhancing social interaction between its users bysuggesting similarities that would otherwise go unnoticed.

REFERENCES

[1] P. KR, R. Mukhopadhyay, J. Philip, A. Jha, V. Namboodiri, and C. Jawa-har, “Towards automatic face-to-face translation,” in Proceedings of the27th ACM International Conference on Multimedia, 2019, pp. 1428–1436.

[2] Q. Mao, B. Feng, and S. Pan, “Modeling user interests using topic model,”Journal of Theoretical and Applied Information Technology, vol. 48, no. 1,pp. 600–606, 2013.

[3] R. Rehurek and P. Sojka, “Software Framework for Topic Modelling withLarge Corpora,” in Proceedings of the LREC 2010 Workshop on NewChallenges for NLP Frameworks. Valletta, Malta: ELRA, May 2010,pp. 45–50.

[4] L. Leydesdorff and K. Welbers, “The semantic mapping of words and co-words in contexts,” Journal of Informetrics, vol. 5, no. 3, pp. 469–475,2011.

[5] K. Sasaki, T. Yoshikawa, and T. Furuhashi, “Twitter-TTM: An efficientonline topic modeling for Twitter considering dynamics of user interestsand topic trends,” 2014 Joint 7th International Conference on SoftComputing and Intelligent Systems, SCIS 2014 and 15th InternationalSymposium on Advanced Intelligent Systems, ISIS 2014, pp. 440–445,2014.

[6] E. Fast, B. Chen, and M. S. Bernstein, “Empath: Understanding topicsignals in large-scale text,” Conference on Human Factors in ComputingSystems - Proceedings, pp. 4647–4657, 2016.

[7] R. Speer, J. Chin, and C. Havasi, “Conceptnet 5.5: An open multilingualgraph of general knowledge,” 2017, pp. 4444–4451. [Online]. Available:http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14972

[8] B. Eisner, T. Rocktaschel, I. Augenstein, M. Bosnjak, and S. Riedel,“emoji2vec: Learning Emoji Representations from their Description,” pp.48–54, 2016.

[9] E. Loper and S. Bird, “Nltk: The natural language toolkit,” in Proceedingsof the ACL-02 Workshop on Effective Tools and Methodologies forTeaching Natural Language Processing and Computational Linguistics,2002, pp. 63–70.