csci 590 : machine learning lower dimensional latent...
TRANSCRIPT
Models
CSCI 590 : Machine Learning
Lower Dimensional Latent Semantic Space
LDA,PLSI,NMF
Halid Ziya Yerebakan
March 23 2015
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
Document-Term Matrices
Figure: Document-term matrix
Document is vector offrequencies of the words.
Mostly sparse matrices.
Bag of words assumption,order information ignored.
Other contexts : Imagepixel , haplotypes , movierating.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
Latent Semantic Space
Figure: Latent Semantic Space
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
Example
Figure: LDA on TREC AP Corpus [1]
iupui
Models
De�nitions
De�nition
Topic : Topic is a distribution over words.
Figure: Topic distribution
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
De�nitions
De�nition
Topic proportions : Mixture weights of topics for a document.Documents have multiple topics.
Figure: Topic proportionsHalid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Outline
1 ModelsPLSILDANMFImplementations
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
PLSI References
Hofman, T. (1999, August). Probabilistic latent semanticindexing. In Proceedings of the 22nd annual international ACMSIGIR conference on Research and development in informationretrieval (pp. 50-57). ACM.
Hofmann, T. (1999, July). Probabilistic latent semanticanalysis. In Proceedings of the Fifteenth conference onUncertainty in arti�cial intelligence (pp. 289-296). MorganKaufmann Publishers Inc..
Hofmann, T. (2001). Unsupervised learning by probabilisticlatent semantic analysis. Machine learning, 42(1-2), 177-196.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
PLSI Neighbors
Figure: PLSI Concept Graph
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Graphical Generative Model
Figure: PLSI Model
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Generative Model
Figure: PLSI Model
Generative Process
1 Select document with probabilityP(d)
2 Pick a latent class (topic) z withprobability P(z|d)
3 Generate a word w withprobability P(w|z)
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
Results
Topics
Figure: Few Topics in PLSI[1]
IR Performance
Figure: PLSI IR performance[1]
iupui
Models
PLSILDANMFImplementations
Advantages - Disadvantages
Advantages
De�nes proper probability distributions on words
More realistic document model, interpretable topics.
Disadvantages
K is �xed
Local Maximum , Over�tting
PLSI is not nested like LSA ( n dimensional solution includesn-1 dimensional solution).
Slow training speed compared to LSA
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Advantages - Disadvantages
Advantages
De�nes proper probability distributions on words
More realistic document model, interpretable topics.
Disadvantages
K is �xed
Local Maximum , Over�tting
PLSI is not nested like LSA ( n dimensional solution includesn-1 dimensional solution).
Slow training speed compared to LSA
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Outline
1 ModelsPLSILDANMFImplementations
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
LDA References
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichletallocation. the Journal of machine Learning research, 3,993-1022.
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2001). Latent dirichletallocation. In Advances in neural information processingsystems.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
LDA Concept Graph
Figure: LDA Concept Graph
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
LDA Model
Figure: LDA Model
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
LDA Model
1 For each document2 Chose N∼Poisson(.)
1 Chose θ ∼ Dir(α)2 For each word wn
1 Chose topic zn ∼ Multinomial(θ)
2 Chose a word from wn ∼ p(wn|βzn )
P(θ, z,w|α.β) = P(θ|α)N∏
n=1
P(zn|θ)P(wn|zn, β)
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
LDA Geometry
[1]
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
Experiments
Figure: LDA on TREC AP Corpus [1]
iupui
Models
PLSILDANMFImplementations
Advantages-Disadvantages
Advantages
De�nes a proper generative model on documents.
It can be incorporated in more complex models
Disadvantages
K is �xed
IID generative assumption of topics.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Advantages-Disadvantages
Advantages
De�nes a proper generative model on documents.
It can be incorporated in more complex models
Disadvantages
K is �xed
IID generative assumption of topics.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Outline
1 ModelsPLSILDANMFImplementations
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
References
Lee, D. D., & Seung, H. S. (1999). Learning the parts ofobjects by non-negative matrix factorization. Nature,401(6755), 788-791.
Seung, D., & Lee, L. (2001). Algorithms for non-negativematrix factorization. Advances in neural information processingsystems, 13, 556-562.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Concept Graph
Figure: NMF Concept Graph
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Non Negative Matrix Factorization
Figure: NMF
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Motivation
Matrix Factorization
Viµ ≈ (WH)iµ =∑r
a=1WiaHaµ
Designed for
Part based representation , no cancellation.
Compressed form of data (n+m)r<nm
Intuitive notions of basis vectors
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Motivation
Matrix Factorization
Viµ ≈ (WH)iµ =∑r
a=1WiaHaµ
Designed for
Part based representation , no cancellation.
Compressed form of data (n+m)r<nm
Intuitive notions of basis vectors
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Algorithm
Objective Function
F =n∑
i=1
m∑µ=1
[Viµlog(WH)iµ − (WH)iµ]
s.t Wia ≥ 0,Haµ ≥ 0
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Algorithm
Update Equations
Wia ← Wia
∑µ
Viµ
(WH)iµHaµ
Wia ←Wia∑j Wja
Haµ ← Haµ
∑i
WiaViµ
(WH)iµ
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Face Images
Figure: NMF,VQ,PCA on face images [1]
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Advantages-Disadvantages
Advantages
Part based representations , intuitive
Sparse encoding
Disadvantages
Di�erent viewpoints or articulated objects cannot be learned.
No learning of syntactic relation between parts.
K is �xed.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Advantages-Disadvantages
Advantages
Part based representations , intuitive
Sparse encoding
Disadvantages
Di�erent viewpoints or articulated objects cannot be learned.
No learning of syntactic relation between parts.
K is �xed.
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Outline
1 ModelsPLSILDANMFImplementations
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space
iupui
Models
PLSILDANMFImplementations
Code
http://www.nltk.org/
Topic Model Toolbox
Mallet
Matlab nnmf
Blei's implementations
Halid Ziya Yerebakan CSCI 590 : Machine Learning Lower Dimensional Latent Semantic Space