topic models for corpora and for graphs. motivation social graphs seem to have –some aspects of...
TRANSCRIPT
![Page 1: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/1.jpg)
Topic models for corpora and for graphs
![Page 2: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/2.jpg)
Motivation• Social graphs seem to have – some aspects of randomness• small diameter, giant connected components,..– some structure• homophily, scale-free degree dist?
![Page 3: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/3.jpg)
More terms• “Stochastic block model”, aka “Block-stochastic matrix”:– Draw ni nodes in block i– With probability pij, connect pairs (u,v) where u is in block i, v is in block j– Special, simple case: pii=qi, and pij=s for all i≠j• Question: can you fit this model to a graph?– find each pij and latent nodeblock mapping
![Page 4: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/4.jpg)
Not? football
![Page 5: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/5.jpg)
Not? books
![Page 6: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/6.jpg)
Outline• Stochastic block models & inference question• Review of text models– Mixture of multinomials & EM– LDA and Gibbs (or variational EM)• Block models and inference• Mixed-membership block models• Multinomial block models and inference w/ Gibbs
![Page 7: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/7.jpg)
Review – supervised Naïve Bayes• Naïve Bayes Model: Compact representationC
W1 W2 W3 ….. WN
C
W
N
M
M
![Page 8: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/8.jpg)
Review – supervised Naïve Bayes• Multinomial Naïve BayesC
W1 W2 W3 ….. WN
M
• For each document d = 1,, M
• Generate Cd ~ Mult( | )
• For each position n = 1,, Nd
• Generate wn ~ Mult( |,Cd)
![Page 9: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/9.jpg)
Review – supervised Naïve Bayes• Multinomial naïve Bayes: Learning– Maximize the log-likelihood of observed variables w.r.t. the parameters:• Convex function: global optimum• Solution:
![Page 10: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/10.jpg)
Review – unsupervised Naïve Bayes• Mixture model: unsupervised naïve Bayes modelC
W
NM
• Joint probability of words and classes:
• But classes are not visible:Z
![Page 11: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/11.jpg)
LDA
![Page 12: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/12.jpg)
Review - LDA• Motivation
w
M
N
Assumptions: 1) documents are i.i.d 2) within a document, words are i.i.d. (bag of words)
•For each document d = 1,,M
• Generate d ~ D1(…)
• For each word n = 1,, Nd
•generate wn ~ D2( | θdn)
Now pick your favorite distributions for D1, D2
![Page 13: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/13.jpg)
• Latent Dirichlet Allocation
z
w
M
N
• For each document d = 1,,M
• Generate d ~ Dir( | )
• For each position n = 1,, Nd
• generate zn ~ Mult( | d)
• generate wn ~ Mult( | zn)
“Mixed membership”kk
jjk nn
nnnnjz
...),...,,|Pr(
11,21
K
![Page 14: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/14.jpg)
• vs Naïve Bayes…
z
w
M
N
K
![Page 15: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/15.jpg)
• LDA’s view of a document
![Page 16: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/16.jpg)
• LDA topics
![Page 17: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/17.jpg)
Review - LDA• Latent Dirichlet Allocation– Parameter learning:• Variational EM– Numerical approximation using lower-bounds– Results in biased solutions– Convergence has numerical guarantees• Gibbs Sampling – Stochastic simulation– unbiased solutions– Stochastic convergence
![Page 18: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/18.jpg)
Review - LDA• Gibbs sampling– Applicable when joint distribution is hard to evaluate but conditional distribution is known– Sequence of samples comprises a Markov Chain– Stationary distribution of the chain is the joint distribution Key capability: estimate
distribution of one latent variables given the other latent variables and observed variables.
![Page 19: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/19.jpg)
Why does Gibbs sampling work?• What’s the fixed point?– Stationary distribution of the chain is the joint distribution• When will it converge (in the limit)?– Graph defined by the chain is connected• How long will it take to converge?– Depends on second eigenvector of that graph
![Page 20: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/20.jpg)
![Page 21: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/21.jpg)
Called “collapsed Gibbs sampling” since you’ve marginalized away some variables
Fr: Parameter estimation for text analysis - Gregor Heinrich
![Page 22: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/22.jpg)
Review - LDA• Latent Dirichlet Allocationz
w
M
N
• Randomly initialize each zm,n
• Repeat for t=1,….
• For each doc m, word n
• Find Pr(zmn=k|other z’s)
• Sample zmn according to that distr.
“Mixed membership”
![Page 23: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/23.jpg)
Outline• Stochastic block models & inference question• Review of text models– Mixture of multinomials & EM– LDA and Gibbs (or variational EM)• Block models and inference• Mixed-membership block models• Multinomial block models and inference w/ Gibbs• Beastiary of other probabilistic graph models– Latent-space models, exchangeable graphs, p1, ERGM
![Page 24: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/24.jpg)
Review - LDA• Motivation
w
M
N
Assumptions: 1) documents are i.i.d 2) within a document, words are i.i.d. (bag of words)
•For each document d = 1,,M
• Generate d ~ D1(…)
• For each word n = 1,, Nd
•generate wn ~ D2( | θdn)
Docs and words are exchangeable.
![Page 25: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/25.jpg)
Stochastic Block models: assume 1) nodes w/in a block z and 2) edges between blocks zp,zq are exchangeable
zp zq
apq
N2
zp
N
p
![Page 26: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/26.jpg)
Stochastic Block models: assume 1) nodes w/in a block z and 2) edges between blocks zp,zq are exchangeable
zp zq
apq
N2
zp
N
p
Gibbs sampling:
• Randomly initialize zp for each node p.
• For t = 1…
• For each node p
• Compute zp given other z’s
• Sample zp
See: Snijders & Nowicki, 1997, Estimation and Prediction for Stochastic Blockmodels for Groups with Latent Graph Structure
![Page 27: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/27.jpg)
Mixed Membership Stochastic Block modelsp q
zp. z.q
apq
N2
p
N
p
Airoldi et al, JMLR 2008
![Page 28: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/28.jpg)
Parkkinen et al paper
![Page 29: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/29.jpg)
Another mixed membership block model
![Page 30: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/30.jpg)
Another mixed membership block modelz=(zi,zj) is a pair of block ids
nz = #pairs z
qz1,i = #links to i from block z1
qz1,. = #outlinks in block z1
δ = indicator for diagonal
M = #nodes
![Page 31: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/31.jpg)
Another mixed membership block model
![Page 32: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/32.jpg)
Experiments
Balasubramanyan, Lin, Cohen, NIPS w/s 2010
+ lots of synthetic data
![Page 33: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/33.jpg)
![Page 34: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/34.jpg)
![Page 35: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/35.jpg)
Experiments
Balasubramanyan, Lin, Cohen, NIPS w/s 2010
![Page 36: Topic models for corpora and for graphs. Motivation Social graphs seem to have –some aspects of randomness small diameter, giant connected components,](https://reader030.vdocuments.us/reader030/viewer/2022032523/56649d775503460f94a58d4e/html5/thumbnails/36.jpg)
Experiments
Balasubramanyan, Lin, Cohen, NIPS w/s 2010