Download - Expectation Maximization Dekang Lin Department of Computing Science University of Alberta
Expectation Maximization
Dekang Lin
Department of Computing Science
University of Alberta
Objectives
Expectation Maximization (EM) is perhaps most often used and mostly half understood algorithm for unsupervised learning. It is very intuitive. Many people rely on their intuition to apply the
algorithm in different problem domains.
I will present a proof of the EM Theorem that explains why the algorithm works. Hopefully this will help applying EM when intuition is
not obvious.
Model Building with Partial Observations
Our goal is to build a probabilistic model A model is defined by a set of parameters θ
The model parameters can be estimated from a set of training examples: x1, x2, …, xn
xi’s are identically and independently distributed (iid)
Unfortunately, we only get to observe part of each training example: xi=(ti, yi) and we can only observe yi.
How do we build the model?
Example: POS Tagging
Complete data: A sentence (a sequence of words) and a corresponding sequence of POS tags.
Observed data: the sentence
Unobserved data: the sequence of tags
Model: an HMM with transition/emission probability tables.
Training with Tagged Corpus
Pierre NNP Vinken NNP , , 61 CD years NNS old JJ , , will MD join VB the DT board NN as IN a DT nonexecutive JJ director NN Nov. NNP 29 CD . .Mr. NNP Vinken NNP is VBZ chairman NN of IN Elsevier NNP N.V. NNP , , the DT Dutch NNP publishing VBG group NN . .Rudolph NNP Agnew NNP , , 55 CD years NNS old JJ and CC former JJ chairman NN of IN Consolidated NNP Gold NNP Fields NNP PLC NNP , , was VBD named VBN a DT nonexecutive JJ director NN of IN this DT British JJ industrial JJ conglomerate NN . .
Pierre NNP Vinken NNP , , 61 CD years NNS old JJ , , will MD join VB the DT board NN as IN a DT nonexecutive JJ director NN Nov. NNP 29 CD . .Mr. NNP Vinken NNP is VBZ chairman NN of IN Elsevier NNP N.V. NNP , , the DT Dutch NNP publishing VBG group NN . .Rudolph NNP Agnew NNP , , 55 CD years NNS old JJ and CC former JJ chairman NN of IN Consolidated NNP Gold NNP Fields NNP PLC NNP , , was VBD named VBN a DT nonexecutive JJ director NN of IN this DT British JJ industrial JJ conglomerate NN . .
c(JJ)=7 c(JJ, NN)=4, P(NN|JJ)=4/7
What is the best Model?
There are many possibly models Many possible ways to set the model parameters.
We obviously want the “best” model.
Which model is the best? The model that assigns the highest probability to the
observation is the best.
Maximize Πi Pθ(yi), or equivalently Σi log Pθ(yi) What about maximizing the probability of the hidden data?
This is know as the maximum likelihood estimation (MLE)
MLE ExampleA coin with P(H)=p, P(T)=q. We observed m H’s and n T’s. What are p and q according to MLE?
Maximize Σi log Pθ(yi)= log pmqn
Under the constraint: p+q=1
Lagrange Method: Define g(p,q)=m log p + n log q+λ(p+q-1) Solve the equations
1 ,0),(
,0),(
qpq
qpg
p
qpg
Example
Suppose we have two coins. Coin 1 is fair. Coin 2 has probability p generating H. They each have ½ probability to be chosen and tossed. The complete data is (1, H), (1, T), (2, T), (1, H), (2, T) We only know the result of the toss, but don’t know when
coin was chosen. The observed data is H, T, T, H, T.
Problem: Suppose the observations include m H’s and n T’s. How to estimate p to maximize Σi log Pθ(yi)?
Need for Iterative Algorithm
Unfortunately, we often cannot find the best θ by solving equations.Example: Three coins, 0, 1, and 2, with probabilities p0, p1, and p2
generating H. Experiment: Toss coin 0
If H, toss coin 1 three times If T, toss coin 2 three times
Observations: <HHH>, <TTT>, <HHH>, <TTT>, <HHH>
What is MLE for p0, p1, and p2?
Overview of EM
Create an initial model, θ0. Arbitrarily, randomly, or with a small set of training
examples.
Use the model θ’ to obtain another model θ such that
Σi log Pθ(yi) > Σi log Pθ’(yi)
Repeat the above step until reaching a local maximum. Guaranteed to find a better model after each iteration.
Maximizing Likelihood
How do we find a better model θ given a model θ’?
Can we use Lagrange method to maximize
ΣilogPθ(yi)? If this can be done, there is no need to iterate!
EM Theorem
The following EM Theorem holds This theorem is similar to (but is not identical to, nor
does it follow) the EM Theorem in [Jelinek 1997, p.148] (the proof is almost identical).
EM Theorem:
Σt is summation over all possible values of unobserved data
ii
ii
i tii
i tii
yPyP
ytPytPytPytP
)(log)(log
),(log)|(),(log)|(
What does EM Theorem Mean?
If we can find a θ that maximizes
the same θ will also satisfy the condition
which is needed in the EM algorithm.We can maximize the former by taking its partial derivatives w.r.t. parameters in θ.
i t
ii ytPytP ),(log)|(
i
ii
i yPyP )(log)(log
ii
ii
i tii
i tii
yPyP
ytPytPytPytP
)(log)(log
),(log)|(),(log)|(
EM Theorem: why?
Why optimizing
is easier than optimizing
Pθ(t, yi) involves the complete data and is usually a product of a set of parameters. Pθ(yi) usually involves summation over all hidden variables.
i t
ii ytPytP ),(log)|(
i
iyP )(log
EM Theorem: Proof
i tii
i tii
i t i
i
i t i
ii
i t i
ii
i t i
ii
i t i
ii
i i t i
iii
t i
iii
ii
ii
ytPytPytPytP
ytP
ytPytP
ytP
ytPytP
ytP
ytPytP
ytP
ytPytP
ytP
ytPytP
ytP
ytPyPytP
ytP
ytPyPytP
yPyP
),(log)|(),(log)|(
),(
),(log)|(
)|(
)|(log)|(
),(
),(log)|(
)|(
),(log)|(
)|(
),(log)|(
),(
),()(log)|(
),(
),()(log)|(
)(log)(log
=1
≤0 (Jensen’s Inequality)
The proof used the inequality
More generally, if p and q are probability distributions
Even more generally, if f is a convex function, E[f(x)] ≥ f(E[x]) Jensen’s Inequality
0)|(
)|(log)|(
t i
ii ytP
ytPytP
0log
x xp
xqxp
What is ?
The expected value of log Pθ(t,yi) according to the model θ’.
The EM Theorem states that we can get a better model by maximizing the sum (over all instances) of the expectation.
t
ii ytPytP ),(log)|(
A Generic Set Up for EM
Assume Pθ(t, y) is a product of a set of parameters.
Assume θ consists of M groups of parameters. The parameters in each group sum up to 1.
Let ujk be a parameter. Σmujm=1
Let Tjk be a subset of hidden data such that if t is in Tjk, the computation of Pθ(t, yi) involves ujk.
Let n(t,yi) be the number of times ujk is used in Pθ(t,yi), i.e., Pθ(t,yi)=ujk
n(t,yi) v(t,y), where v(t,y) is the product of all other
parameters.
j
i Ttii
jk
jjk
ii Tt
i
jjk
i Tti
ytnjki
jk
i t l mlmlii
jk
jk
jk
i
ytnytP
u
u
ytnytP
u
ytvuytP
u
uytPytP
),()|(
0
),()|(
),(log)|(
1),(log)|(
),(
pseudo count of instancesinvolving ujk
Summary
EM Theorem Intuition Proof
Generic Set-up