- 1 -

Bayesian inference of normal distribution

• Problem statement– Objective is to estimate unknown two parameters q={ ,m s2} of normal

distribution based on observations y = {y1, y2, …}.

• Prior– Joint pdf of non-informative prior

• Joint posterior distribution

• This is function of two variables, which can be plotted as a surface or contour.

12 2,p

22 2 22

1, | exp 1

2np y n s n y

222

11

2 2 22 22 2

1 1

1| | exp

2

1 1exp exp

2 2

n nn

i iii

n nn n

i ii i

p y p y y

y y y y y n y

22 1where

1 is yn

- 2 -

Joint posterior distribution

• Distribution

– There is no inherent pdf function by matlab in this case.• This is function of two variables, which can be plotted as a surface or contour.

– Let’s consider a case with n=20; y =2.9; s=0.2;

• Remark– Analysis of posterior pdf: mean, median & confidence bounds.

• Marginal distribution

• Once we have the marginal pdf, we can evaluate its mean and confidence bounds.

– Posterior prediction: predictive distribution of new y based on observed y.

22 2 22

1, | exp 1

2np y n s n y

2 2 2| | , , |p y y p y p y d d

2 2 2 2| , | , | , |p y p y d p y p y d

This is posterior pdf of & m s2 This is likelihood of new y

- 3 -

• Analytical solutions

– Marginal mean

– Marginal variance

• How to evaluate marginal distributions– Before doing this, note that in normal dist., variable y can be transformed to

standard normal variable z.

– Marginal mean & variance can be evaluated in the same manner.– One can evaluate various characteristics using the matlab functions.

Marginal distributions

2

2 2 22| , | 1 ~ 1s

p y p y d n n

2 2| , |p y p y d

~ , , ~ 0,1y

y N z z N

, ~ 1/

yz z t ns n

2

2 221 , ~ 1s

z n z n

22 2 22

1, | exp 1

2np y n s n y

2

21

2

z

f z e

- 4 -

Marginal distributions

• Supplementary information for normal pdf– The original pdf fY(y) and normalized pdf fZ(z) have following relation.

– Therefore, – If we want use f(z) instead of f(y), use this relation.

• Supplementary information for chi2 pdf– The original pdf fY(y) and normalized pdf fZ(z) have following relation.

– Therefore,

1Y Zf y f z

z y y y

Zz Z Z Y y

f zyF f z dz f z d dy f y dy F

22

2Z

zf f z

2

2 22

2 2 22

b a

a b

z

z Z Zz

zF f z dz f z d f d F

- 5 -

Posterior prediction

• Predictive distribution of new y based on observed y

• Analytical solution

– One can evaluate characteristics of this using the matlab function.– Compare with the marginal mean.

• Marginal mean: t-distribution with location y and variance s^2/n with n-1 dof.• Predictive distribution: t-distribution with location y and variance s^2*(1+1/n) with n-1

dof.

2 2 2| | , , |p y y p y p y d d

posterior pdf of & m s2 Likelihood of new y

22 2 22

1, | exp 1

2np y n s n y

22

2

1| , exp

22

yp y

| ~ 11

1

y yp y y z z t n

sn

- 6 -

Simulation of joint posterior distribution

• Why ?– Even when analytic solutions available, some are not easy to evaluate.– Once exercised, may find it more convenient and more general.– Practice simulation & validate with analytic solution.

• Factorization approach

– Review: joint probability of A & B

– Likewise, joint posterior pdf of & m s2

• The two at the right are marginal pdfs.

• Simulation by drawing random samples– We can evaluate characteristics of joint posterior pdf using simulation

techniques.

S

A

B

Pr(A | B)

, | |P A B P A B P B P B A P A

2 2 2 2, | | , | | , |p y p y p y p y p y

- 7 -

Simulation of joint posterior distribution

• Approach 1: use marginal variance.

– Conditional pdf of m on s2 is already derived, which is the pdf of mean with known variance. (page 10 of Lec. #4)

– Marginal pdf of s2 is given in this lecture.

– In order to sample the posterior pdf of p( ,m s2|y)

1. Draw s2 from the marginal pdf

2. Draw m from the conditional pdf

2 2 2, | | , |p y p y p y

2 2| ~ , /N y n

2 2 2| ~ Inv 1, or y n s 2

22

1 ~ 1s

z n n

2 2 2| ~ Inv 1,y n s

2 2| ~ , /N y n

Marginal pdf of s2Conditional pdf of m on s2

- 8 -

Simulation of joint posterior distribution

• Approach 1: use marginal variance.

– Once you have obtained samples of joint pdf, compare (validate) the results with the analytic solution.

1. Compare the samples of (M,S2) with the analytic joint pdf.• In terms of scattered pot & contour.• In terms of 3-D histogram & surface (mesh).

2. Compare the samples of M with the marginal pdf, which is t distribution.

3. Compare the samples of S2 with the marginal pdf which is inv-chi2 dis-tribution.

4. Extract features of the samples M and compare with analytic solution.

5. Extract features of the samples S2 and compare with analytic solution.

- 9 -

Simulation of joint posterior distribution

• Approach 2: use marginal mean.

– Conditional pdf of s2 on m is already derived, which is the pdf of vari-ance with known mean. (page 13 of Lec. #4)

– Marginal pdf of m is given in this lecture.

– In order to sample the posterior pdf of p( ,m s2|y)

1. Draw m from the marginal pdf

2. Draw s2 from the conditional pdf

– Compare (validate) the resulting samples with the analytic solution.

Marginal pdf of mConditional pdf of s2 on m

2 2, | | , |p y p y p y

2 2| ~ Inv , orn v 2

22 22

1~ where 1z n n v n s n y

n

21| ~ , / or ny t y s n ~ 1

/

yz t ns n

21| ~ , /ny t y s n

2 2| ~ Inv ,n v

- 10 -

Simulation of joint posterior distribution

• Approach 2: use marginal mean.

– Once you have obtained samples of joint pdf, compare (validate) the results with the analytic solution.

– Details are omitted for limited time.

- 11 -

Posterior prediction by simulation

• Analytic approach

– Result of analytic solution by double integral over infinity to get

• Simulation– In practice, posterior predictive distribution is obtained by random

draws. – Once we have posterior distribution for & m s2 in the form of samples,

the predictive new y are easily obtained by drawing each one from conditional on each individual & m s2.

– Mean & conf. intervals of posterior prediction can be obtained easily.

2 2 2| | , , |p y y p y p y d d

22 2 22

1, | exp 1

2np y n s n y

22

2

1| , exp

22

yp y

| ~ 11

1

y yp y y z z t n

sn

2| ,p y

- 12 -

Homework

- 13 -

Homework

- 14 -

Homework

• Problems 1

1) Use the approach 1 to obtain samples of joint pdf (M,S2).2) Compare scattered dots with contour of analytic solution.

Compare 3-D histogram with mesh shape of analytic solution.

3) Compare the samples of M with the marginal pdf, which is t distribution.

4) Compare the samples of S2 with the marginal pdf which is inv-chi2 dist.

5) Obtain mean, 95% conf. interval of M, & compare with analytic solution.

6) Obtain mean, 95% conf. interval of S2, & compare with analytic solu-tion.

7) Obtain samples of posterior prediction, and obtain mean, 95% conf. interval of ynew & compare with analytic solu-tion.