ECE 8443 – Pattern RecognitionECE 8527 – Introduction to Machine Learning and Pattern Recognition

• Objectives:Jensen’s Inequality (Special Case)EM Theorem ProofEM Example – Missing DataApplication: Hidden Markov Models

• Resources:Wiki: EM HistoryT.D.: Brown CS TutorialUIUC: TutorialF.J.: Statistical Methods

LECTURE 10: EXPECTATION MAXIMIZATION (EM)

ECE 8527: Lecture 10, Slide 2

The Expectation Maximization Algorithm (Preview)

ECE 8527: Lecture 10, Slide 3

The Expectation Maximization Algorithm (Cont.)

ECE 8527: Lecture 10, Slide 4

The Expectation Maximization Algorithm

ECE 8527: Lecture 10, Slide 5

• Expectation maximization (EM) is an approach that is used in many ways to find maximum likelihood estimates of parameters in probabilistic models.

• EM is an iterative optimization method to estimate some unknown parameters given measurement data. Used in a variety of contexts to estimate missing data or discover hidden variables.

• The intuition behind EM is an old one: alternate between estimating the unknowns and the hidden variables. This idea has been around for a long time. However, in 1977, Dempster, et al., proved convergence and explained the relationship to maximum likelihood estimation.

• EM alternates between performing an expectation (E) step, which computes an expectation of the likelihood by including the latent variables as if they were observed, and a maximization (M) step, which computes the maximum likelihood estimates of the parameters by maximizing the expected likelihood found on the E step. The parameters found on the M step are then used to begin another E step, and the process is repeated.

• This approach is the cornerstone of important algorithms such as hidden Markov modeling and discriminative training, and has been applied to fields including human language technology and image processing.

Synopsis

ECE 8527: Lecture 10, Slide 6

Lemma: If p(x) and q(x) are two discrete probability distributions, then:

with equality if and only if p(x) = q(x) for all x.

Proof:

The last step follows using a bound for the natural logarithm: .

Special Case of Jensen’s Inequality

xx

xqxpxpxp )(log)()(log)(

xx

x

x

x

xx

xpxqxp

xpxqxp

xpxqxp

xqxpxp

xqxpxp

xqxpxpxp

)1)()()((

)()(log)(

0)()(log)(

0)()(log)(

0)()(log)(

0)(log)()(log)(

1ln xx

ECE 8527: Lecture 10, Slide 7

Continuing in efforts to simplify:

We note that since both of these functions are probability distributions, they must sum to 1.0. Therefore, the inequality holds.

The general form of Jensen’s inequality relates a convex function of an integral to the integral of the convex function and is used extensively in information theory.

Special Case of Jensen’s Inequality

x x xxxx

xpxqxpxpxqxp

xpxqxp

xpxqxp ..0)()()(

)()()()1

)()()((

)()(log)(

ECE 8527: Lecture 10, Slide 8

Theorem: If then .

Proof: Let y denote observable data. Let be the probability distribution

of y under some model whose parameters are denoted by .

Let be the corresponding distribution under a different setting .

Our goal is to prove that y is more likely under than .

Let t denote some hidden, or latent, parameters that are governed by the

values of . Because is a probability distribution that sums to 1, we

can write:

Because we can exploit the dependence of y on t and using well-known

properties of a conditional probability distribution.

The EM Theorem

ytPytPytPytPtt

loglog yPyP

yP

yP

ytP

tt

yPytPyPytPyPyP loglogloglog

ECE 8527: Lecture 10, Slide 9

We can multiply each term by “1”:

where the inequality follows from our lemma.

Explanation: What exactly have we shown? If the last quantity is greater than

zero, then the new model will be better than the old model. This suggests a

strategy for finding the new parameters, θ: choose them to make the last

quantity positive!

Proof Of The EM Theorem

tt

tt

tt

tt

tt

ytPytPytPytP

ytPytPytPytP

ytPytPytPytP

ytPytPytP

ytPytPytP

ytPytPyPytP

ytPytPyPytPyPyP

,log,log

,loglog

,log,log

,log,log

,,log

,,logloglog

ECE 8527: Lecture 10, Slide 10

Discussion

• If we start with the parameter setting , and find a parameter setting for

which our inequality holds, then the observed data, y, will be more probable

under than .

• The name Expectation Maximization comes about because we take the

expectation of with respect to the old distribution and then

maximize the expectation as a function of the argument .

• Critical to the success of the algorithm is the choice of the proper

intermediate variable, t, that will allow finding the maximum of the

expectation of .

• Perhaps the most prominent use of the EM algorithm in pattern recognition is

to derive the Baum-Welch reestimation equations for a hidden Markov model.

• Many other reestimation algorithms have been derived using this approach.

ytP , ytP ,

ytPytPt

log

ECE 8527: Lecture 10, Slide 11

Example: Estimating Missing Data

2*,

22

,01,

20

,,, 4321 xxxxD

222121 Tθ

• Consider a data set with a missing element:

• Let us estimate the value of the missing point assuming a Gaussian model

with a diagonal covariance and arbitrary means:

• Expectation step:

Assuming normal distributions as initial conditions, this can be simplified to:

ytPytPt

log

41

4141

41

413

1

41420

413

14

4

44

ln(nl

)4;((ln(ln);(

dxxd

xp

xp

xpp

dxxxpppQ

kk

kk

0

0

θ

θθθx

θθxθxθθ

3

1212

2

22

21

21 )2ln(

2)4(

21(ln);(

kkpQ

θxθθ

ECE 8527: Lecture 10, Slide 12

Example: Gaussian Mixtures

• An excellent tutorial on Gaussian mixture estimation can be found at

J. Bilmes, EM Estimation

• An interactive demo showing convergence of the estimate can be found at

I. Dinov, Demonstration

ECE 8527: Lecture 10, Slide 13

Introduction To Hidden Markov Models

ECE 8527: Lecture 10, Slide 14

Introduction To Hidden Markov Models (Cont.)

ECE 8527: Lecture 10, Slide 15

Introduction To Hidden Markov Models (Cont.)

ECE 8527: Lecture 10, Slide 16

Summary• Expectation Maximization (EM) Algorithm: a generalization of Maximum

Likelihood Estimation (MLE) based on maximization of a posterior that data was generated by a model. EM is a special case of Jensen’s inequality.

• Jensen’s Inequality: describes a relationship between two probability distributions in terms of an entropy-like quantity. A key tool in proving that EM estimation converges.

• The EM Theorem: proved that estimation of a model’s parameters using an iteration of EM increases the posterior probability that the data was generated by the model.

• Demonstrated an application of the EM Theorem to the problem of estimating missing data point.

• Explained how EM can be used to reestimate parameters in a pattern recognition system.

• Introduced the concept of a hidden Markov model and explained how we will use EM to estimate the parameters of this model.