Dynamic Embeddings for User Profiling in Twitter

1 KAUST, Saudi Arabia2 JD.com, China 3 University of Amsterdam, The Netherlands

Shangsong Liang1, Xiangliang Zhang1,

Zhaochun Ren2, Evangelos Kanoulas3

2

Overview

Ò The Task Background and Related Work

Ò Our Method

Ò Dynamic User and Word Embedding Model (DUWE)

Ò Streaming Keyword Diversification Model (SKDM)

Ò Experiments

Ò Conclusion

3

The Task

Input: A stream of tweets generated across the time

Output: A set of keywords to profile the user at different point in time

Tweets over timeTwitter Users

Given a user at time t

SportFood

4

The Task

Tweets over timeTwitter Users

Given a user at time t

SportFood

Relevant

Diversified

Dynamic

5

Background of User Profiling Problem

Ò Expert finding task at TREC 2005 enterprise trackÒ Given documents which describes expert candidates, answer

a query with a sorted name list in a specific domain,☛ uncovering associations between people and topics

Ò A generative language modeling approach in Balong et al(2007)

Ò Works on a Static document collectionÒ Assumes users’ profiling results are unchanged

Need Dynamic User Profiling

6

Dynamic User Profiling Approaches

Ò ExperTime (Rybak et al 2014)

Ò A probabilistic model for learning how personal researchinterests evolve (Fang and Godavarthy 2014)

7

Limitations of Current User Profiling Methods

Ò Treat words as atomic units leading to a vocabulary mismatch that harms performance

Ò Represent words and users in disjoint vocabulary spaces making it difficult to measure the similarity between users and words when constructing the profile

Can words and users be embedded in the samesemantic space?

Can their embedding be modeled in the dynamicenvironment?

8

Related Work in Dynamic Topic Models and Dynamic Embedding

Ò Dynamic Topic Models: modeling dynamic user interestsÒ Topic over time model (Wang et al. KDD 2006)Ò Topic tracking model (Iwata et al. IJCAI 2009)Ò Dynamic user clustering topic model (Liang et al. KDD 2016), etcÒ None of them is for user profiling

Ò Dynamic Word EmbeddingÒ Dynamic word embedding by separating data into time bins, and apply

word2vec within each bin (Kim et al. 2014, Hamilton et al. 2016) Ò Or based on Bayesian skip-gram model (Bamler and Mandt, 2017)Ò All of them are for words only but not for usersÒ All of them are not for user profiling

9

Overview

Ò The Task Background and Related Work

Ò Our Method

Ò Dynamic User and Word Embedding Model (DUWE)

Ò Streaming Keyword Diversification Model (SKDM)

Ò Experiments

Ò Conclusion

10

Our Approach

Ò Dynamic User and Word Embedding Model (DUWE)Ò Infer both users’ and words’ embeddings over time in the

same semantic spaceÒ Enable to measure the similarities between users’ and words’

embeddings

Ò Streaming Keyword Diversification ModelÒ Retrieve relevant keywords to profile users’ current interests

over timeÒ Diversify the returned relevant keywords such that the

keywords can cover all aspects of the users’ interests

Dynamic User and Word Embedding

11

vt−1

zt−1

βt−1

ut−1

yt−1

αt−1

vt

zt

βt

ut

yt

αt

n+t−1 m

+t−1 n

+t

m+t

V V |Ut||Ut−1|

Word representation at t-1

User representation at t

Observed co-occurrence of words at t-1

Observed user-word pairs at t-1

p(Ut | Ut�1) / N (Ut�1,↵↵↵2t�1I) · N (0,↵↵↵2

0 I)

p(Vt | Vt�1) / N (Vt�1,���2t�1I) · N (0,���2

0 I)

User Diffusion

Word Diffusion

12

Diffusion of user representation

According to Kalman filtering, we define the variance of transition kernel for auser embedding from t-1 to t

.

p(Ut | Ut�1) / N (Ut�1,↵↵↵2t�1I) · N (0,↵↵↵2

0 I)

GaussianPrior

• A

• F

• F

measuring the word distribution changes from previous time step t-1 to the current time step t for user u

13

Diffusion of word representation

According to Kalman filtering, we define the variance of transition kernel for aword embedding from t-1 to t

.

GaussianPrior

• A

• F

• F

measuring the word distribution changes from t-1 to the current time step t

p(Vt | Vt�1) / N (Vt�1,���2t�1I) · N (0,���2

0 I)

14

DUWE model inference

Ò Apply the skip-gram filtering for the inference (Bamler et al. 2017) and the variational inference algorithm to obtain the embeddings

Ò Posterior distribution over and conditional on the statistics information and as follows:

where we have:

skip-gram model for words skip-gram model for user and words

model transition for users model transition for words

positive and negative indicator matrices for all word-to-word pairs

positive and negative indicator matrices for all user-to-word pairs

15

Streaming Keyword Diversification ModelÒ generating top-K relevant and diversified keywords for

profiling users’ interests at time t.

16

Overview

Ò The Task Background and Related Work

Ò Our Method

Ò Dynamic User and Word Embedding Model (DUWE)

Ò Streaming Keyword Diversification Model (SKDM)

Ò Experiments

Ò Conclusion

17

Experimental Setup

Ò DatasetsÒ 1,375 users randomly sampled from TwitterÒ 3.78 million tweets posted by the users from the beginning of their

registrations up to May 31, 2015Ò Two types of Ground Truth: One for evaluating Relevance-oriented

(RGT) performance and another for evaluating Diversity-oriented (DGT) performance.

Ò Evaluation MetricsÒ Relevance: Pre (Precision), NDCG, MRR, MAPÒ Their semantic version of the metrics, denoted as Pre-S, NDCG-S,

MRR-S, MAP-SÒ Diversity: Pre-IA (Intent-Aware Precision), α-NDCG, MRR-IA, MAP-IA

18

Experimental Setup

Ò BaselinesÒ Non-dynamic Embedding Models

Ò Skip-Gram Model, i.e., word2vec Model (SGM)Ò Distributed Representations of Documents (DRD)

Ò Dynamic Traditional Profiling ModelÒ Predictive Language Model (PLM)

Ò Dynamic Topic ModelÒ User Clustering Topic model (UCT)

Ò Dynamic Embedding ModelsÒ Dynamic Independent Skip-Gram model (DISG)Ò Dynamic Pre-initialized Skip-Gram model (DPSG)Ò Dynamic Independent Distributed Representations of documents

(DIDR)Ò Dynamic Pre-initialized Distributed Representations of documents

(DPDR)

19

Overall Performance

Ò Average relevance performance on time periods of each month

20

Overall Performance

Ò Diversity performance on time periods of each month

An Example User’s Dynamic Profiling Results over Time

21

Top-6 keywords of an example user’s dynamic profile, whose interests cover a number of aspects and dramatically change over time, from Sport, fitness, kitchen, exercise, to education.

Relevance and diversity performance over time

22

Relevance performance over time Diversity performance over time

Performance w.r.t. embedding dimensionality

23

24

Overview

Ò The Task Background and Related Work

Ò Our Method

Ò Dynamic User and Word Embedding Model (DUWE)

Ò Streaming Keyword Diversification Model (SKDM)

Ò Experiments

Ò Conclusion

25

Conclusions

Ò Study the problem of dynamic user profiling in Twitter

Ò Propose a Dynamic User and Word Embedding model (DUWE)

Ò Propose a Streaming Keyword Diversification Model (SKDM)

Ò Evaluate the performance of the proposed models in real dataset, Twitter

Thank you for your attention!

Our paper at

http://www.kdd.org/kdd2018/accepted-papers/view/dynamic-

embeddings-for-user-profiling-in-twitter

Lab of Machine Intelligence and kNowledge Engineering (MINE): http://mine.kaust.edu.sa/