bayesian statistics - an introduction - qmul mathsfjw/goldsmiths/2008/lp/bayestalk.pdf · bayesian...

Bayesian StatisticsStochastic Simulation - Gibbs sampling

Bayesian Statistics - an Introduction

Dr Lawrence Pettit

School of Mathematical Sciences, Queen Mary, University of London

July 22, 2008

Dr Lawrence Pettit Bayesian Statistics - an Introduction

What is Bayesian Statistics?Bayes TheoremThe Likelihood PrincipleMixtures of conjugate priorsPoisson examplePredictive distributionsA little decision theory

What is Bayesian Statistics?

I Based on an idea of subjective probability.

I Have knowledge, beliefs about matter in hand.

I Express these as a (prior) probability distribution.

I Collect some data (likelihood).

I Use Bayes theorem to combine prior knowledge and newinformation to find new (posterior) probability distribution.

I Today’s posterior is tomorrow’s prior.

I Although different people will start with different priors withenough data opinions will converge.

I One coherent paradigm for all problems.

I Can handle more realistic complex models.

I Makes predictions.

I The interpretation of interval estimates is more natural.

I Nuisance parameter and constraints on parameters can beeasily dealt with.

Bayes theorem

I Bayes Theorem is named after Rev. Thomas Bayes, anonconformist minister who lived in England in the first halfof the eighteenth century.

I The theorem was published posthumously in 1763 in ‘Anessay towards solving a problem in the doctrine of chances’.

Bayes theorem

I Bayes Theorem is named after Rev. Thomas Bayes, anonconformist minister who lived in England in the first halfof the eighteenth century.

I The theorem was published posthumously in 1763 in ‘Anessay towards solving a problem in the doctrine of chances’.

Bayes theorem

I Let Ω be a sample space and B1,B2, . . . ,Bk be mutuallyexclusive and exhaustive events in Ω (i.e.Bi ∩ Bj = ∅, i 6= j ,∪k

i=1Bi = Ω; the Bi form a partition of Ω.)

I Let A be any event with Pr[A] > 0.

Pr[Bi |A] =Pr[Bi ] Pr[A|Bi ]

Pr[A]=

Pr[Bi ] Pr[A|Bi ]∑kj=1 Pr[Bj ] Pr[A|Bj ]

Bayes theorem

Pr[A]=

Bayes theorem

Pr[A]=

Example

I A diagnostic test for a disease gives a correct result 99% ofthe time. Suppose 2% of the population have the disease. If aperson selected at random from the population is given thetest and produces a positive result what is the probability thatthe person has the disease? Suppose they are given a secondindependent test and it is also positive, what is the probabilitythat they have the disease now?

I Let ‘+’ and ‘−’ denote the events that a test is positive ornegative, respectively. Let D denote the event that the personhas the disease.

I We require p(D|+) and p(D|+ +).

Example

I A diagnostic test for a disease gives a correct result 99% ofthe time. Suppose 2% of the population have the disease. If aperson selected at random from the population is given thetest and produces a positive result what is the probability thatthe person has the disease? Suppose they are given a secondindependent test and it is also positive, what is the probabilitythat they have the disease now?

I We require p(D|+) and p(D|+ +).

Solution

I We require p(D|+) and p(D|+ +). We are told that

p(+|D) = 0.99, p(−|D) = 0.99, p(D) = 0.02.

I By Bayes theorem

p(D|+) =p(+|D)p(D)

p(+|D)p(D) + p(+|D)p(D)

=0.99× 0.02

0.99× 0.02 + 0.01× 0.98= 0.6689

Solution

I We require p(D|+) and p(D|+ +). We are told thatI

p(+|D) = 0.99, p(−|D) = 0.99, p(D) = 0.02.

I By Bayes theorem

p(D|+) =p(+|D)p(D)

p(+|D)p(D) + p(+|D)p(D)

=0.99× 0.02

0.99× 0.02 + 0.01× 0.98= 0.6689

Solution

I We require p(D|+) and p(D|+ +). We are told thatI

p(+|D) = 0.99, p(−|D) = 0.99, p(D) = 0.02.

I By Bayes theorem

p(D|+) =p(+|D)p(D)

p(+|D)p(D) + p(+|D)p(D)

=0.99× 0.02

0.99× 0.02 + 0.01× 0.98= 0.6689

Solution for a second test

I Suppose that we have a second positive test result

p(D|+ +) =p(+ + |D)p(D)

p(+ + |D)p(D) + p(+ + |D)p(D)

=0.992 × 0.02

0.992 × 0.02 + 0.012 × 0.98= 0.9950

I Thus although the test is very accurate we need two positiveresults before we can say with confidence that the person hasthe disease.

Solution for a second test

I Suppose that we have a second positive test result

p(D|+ +) =p(+ + |D)p(D)

p(+ + |D)p(D) + p(+ + |D)p(D)

=0.992 × 0.02

0.992 × 0.02 + 0.012 × 0.98= 0.9950

I Thus although the test is very accurate we need two positiveresults before we can say with confidence that the person hasthe disease.

Density form of Bayes Theorem

I Let X , θ be two continuous r.v.’s (possibly multivariate).

f (θ|x) =f (θ)f (x |θ)

f (x)=

f (θ)f (x |θ)∫f (θ′)f (x |θ′)dθ′

I Posterior ∝ Likelihood × Prior

p(θ|x) ∝ p(x |θ)p(θ)

f (θ|x) =f (θ)f (x |θ)

f (x)=

f (θ)f (x |θ)∫f (θ′)f (x |θ′)dθ′

f (θ|x) =f (θ)f (x |θ)

f (x)=

f (θ)f (x |θ)∫f (θ′)f (x |θ′)dθ′

The Likelihood Principle

I To illustrate the difference between the classical and Bayesianapproaches we start with an example.

I Suppose we toss a drawing pin and get 9 ‘ups’ and 3 ‘downs’.We denote ‘up’ by U and ‘down’ by D. Is the pin unbiased?

I Classically we might test H0 : p = 12 versus H1 : p > 1

2 wherep = p(U). The probability of the observed result or somethingmore extreme (tail area) if H0 is true is(

which = 2994096 ≈ 7.3%. Thus we would accept H0 at the 5%

level.

I However this assumes that we did the experiment bydeciding to toss the drawing pin 12 times.

I What if we decided to toss the pin until we achieved 3 D’s?I Now the probability of the observed result or something

more extreme if H0 is true is(11

+ · · ·

We may calculate the probability of the complement of thisevent by(

+ · · ·+(

It follows that the P-value is 1344096 ≈ 3.25%. So we would

reject H0 at the 5% level.

I What if we decided to toss the pin until we achieved 3 D’s?

I Now the probability of the observed result or somethingmore extreme if H0 is true is(

+ · · ·

+ · · ·+(

reject H0 at the 5% level.

I What if we decided to toss the pin until we achieved 3 D’s?I Now the probability of the observed result or something

more extreme if H0 is true is(11

+ · · ·

+ · · ·+(

reject H0 at the 5% level.Dr Lawrence Pettit Bayesian Statistics - an Introduction

I In order to perform a significance test we are required tospecify a sample space i.e. the space of all possibleoutcomes.

I Possibilities for the drawing pin are:(i) (u, d) : u + d = 12,(ii) (u, d) : d = 3,or if I carry on tossing the pin until my coffee is ready, sothat there is a random stopping point(iii) all (u, d).

I In order to perform a significance test we are required tospecify a sample space i.e. the space of all possibleoutcomes.

I Possibilities for the drawing pin are:(i) (u, d) : u + d = 12,(ii) (u, d) : d = 3,or if I carry on tossing the pin until my coffee is ready, sothat there is a random stopping point(iii) all (u, d).

I The Bayesian analysis of this problem is somewhat different.Let θ be the chance that the pin lands up.

I θ is a “long run frequency” of U’s. It is an objective propertyof the pin. It does not depend on You.

I You have beliefs about θ which you express in the form of aprobability density function (pdf) p(θ). You use Bayestheorem to update your beliefs.

p(θ|data) ∝ p(data|θ)p(θ)

θ9(1− θ)3p(θ)

The sampling rule is irrelevant.

θ9(1− θ)3p(θ)

I In deriving the posterior distribution the only contributionfrom the data is through the likelihood p(data|θ). Thus aBayesian inference, which will depend only on the posteriordistribution, obeys the likelihood principle.

I This roughly says that if two experiments give the samelikelihoods then the inferences we make should be the same ineach case.

I Classical hypothesis tests or confidence intervals violate thelikelihood principle.

I We need to give a prior distribution for θ.

I Suppose we take

p(θ) =Γ(a + b)

Γ(a)Γ(b)θa−1(1− θ)b−1 a, b > 0

ie a beta distribution with mean a/(a + b) and variance

a + b + 1.

We shall write this distribution as Be(a, b).I It then follows that

p(θ|data) ∝ θ9+a−1(1− θ)3+b−1.

That is Be(9 + a, 3 + b). Thus if we take a beta prior for θ weshall obtain a beta posterior.

I We need to give a prior distribution for θ.I Suppose we take

p(θ) =Γ(a + b)

Γ(a)Γ(b)θa−1(1− θ)b−1 a, b > 0

a + b + 1.

We shall write this distribution as Be(a, b).

I It then follows that

p(θ|data) ∝ θ9+a−1(1− θ)3+b−1.

I We need to give a prior distribution for θ.I Suppose we take

p(θ) =Γ(a + b)

Γ(a)Γ(b)θa−1(1− θ)b−1 a, b > 0

a + b + 1.

We shall write this distribution as Be(a, b).I It then follows that

p(θ|data) ∝ θ9+a−1(1− θ)3+b−1.

I The choice a = b = 1 gives a uniform distribution as the prior,ie we think any value of θ is equally likely. More realisticallyfor a pin we might take a = b = 2, reflecting a belief that wethink θ is more likely to be near 0.5 than 0 or 1 but not beingvery sure.

I Others might choose an asymmetric prior, perhaps arguingthat a pin with a very long point would very likely land downso a pin with any point would land down more often than up. Iam not convinced by this argument but it shows that differentpeople do have different beliefs and one of the advantages ofthe Bayesian approach is that the analysis can reflect these.

I The choice a = b = 1 gives a uniform distribution as the prior,ie we think any value of θ is equally likely. More realisticallyfor a pin we might take a = b = 2, reflecting a belief that wethink θ is more likely to be near 0.5 than 0 or 1 but not beingvery sure.

I Others might choose an asymmetric prior, perhaps arguingthat a pin with a very long point would very likely land downso a pin with any point would land down more often than up. Iam not convinced by this argument but it shows that differentpeople do have different beliefs and one of the advantages ofthe Bayesian approach is that the analysis can reflect these.

I If we were throwing a coin rather than a pin then we wouldalmost certainly choose a different prior. We have much moreexperience throwing coins than pins and are much more surethat θ, the chance the coin will land heads, is close to 0.5.Thus we might take a = b = 50 say.

I The posteriors we get for the pin and the coin will be verydifferent. For the pin the posterior mean will be 11/16 =0.6875 and the posterior variance11/16× 5/16× 1/17 = 0.0126. For the coin the posteriormean will be 59/112 = 0.527 which is close to 0.5.

I The classical unbiased estimate is 9/12 = 0.75 if the numberof throws is fixed, or 9/11 = 0.818 if we continue until wehave three ‘failures’. The classical answers are the samewhether for pins or coins, ignoring the extra information thatwe have.

Plot for pins

Plot for coins

I By taking mixtures of conjugate priors we can represent morerealistic beliefs.

I As an example consider the result when a coin is spun on itsedge. Experience has shown that when spinning a coin theproportion of heads is more likely to be 1/3 or 2/3 than 1/2.Therefore a bimodal prior seems appropriate. Since spinningcoins will be Bernoulli trials the beta distribution will beconjugate.

I Therefore we take the following prior

p(θ) =1

2Be(10, 20) +

2Be(20, 10)

p(θ) =1

Γ(30)

Γ(10)Γ(20)θ10−1(1−θ)20−1+

Γ(30)

Γ(20)Γ(10)θ20−1(1−θ)10−1

where θ is the chance that the coin comes down heads.

p(θ) =1

2Be(10, 20) +

2Be(20, 10)

p(θ) =1

Γ(30)

Γ(10)Γ(20)θ10−1(1−θ)20−1+

Γ(30)

Γ(20)Γ(10)θ20−1(1−θ)10−1

where θ is the chance that the coin comes down heads.

p(θ) =1

2Be(10, 20) +

2Be(20, 10)

p(θ) =1

Γ(30)

Γ(10)Γ(20)θ10−1(1−θ)20−1+

Γ(30)

Γ(20)Γ(10)θ20−1(1−θ)10−1

where θ is the chance that the coin comes down heads.Dr Lawrence Pettit Bayesian Statistics - an Introduction

I If we observe 3 heads and 7 tails in 10 spins the posteriorbased on the first prior is Be(13, 27) and on the second prioris Be(23, 17).

I The weight on the first posterior, α′, is

(0.5p1(x))(0.5p1(x) + 0.5p2(x))−1

p1(x) =Γ(30)

Γ(10)Γ(20)

Γ(13)Γ(27)

Γ(40)

and similarly

p2(x) =Γ(30)

Γ(20)Γ(10)

Γ(23)Γ(17)

Γ(40)

A little calculation shows that α′ = 0.89 and β′ = 0.11.Hence the posterior is

p(θ|x) = 0.89Be(13, 27) + 0.11Be(23, 17)

(0.5p1(x))(0.5p1(x) + 0.5p2(x))−1

p1(x) =Γ(30)

Γ(10)Γ(20)

Γ(13)Γ(27)

Γ(40)

and similarly

p2(x) =Γ(30)

Γ(20)Γ(10)

Γ(23)Γ(17)

Γ(40)

A little calculation shows that α′ = 0.89 and β′ = 0.11.

Hence the posterior is

p(θ|x) = 0.89Be(13, 27) + 0.11Be(23, 17)

(0.5p1(x))(0.5p1(x) + 0.5p2(x))−1

p1(x) =Γ(30)

Γ(10)Γ(20)

Γ(13)Γ(27)

Γ(40)

and similarly

p2(x) =Γ(30)

Γ(20)Γ(10)

Γ(23)Γ(17)

Γ(40)

A little calculation shows that α′ = 0.89 and β′ = 0.11.Hence the posterior is

p(θ|x) = 0.89Be(13, 27) + 0.11Be(23, 17)Dr Lawrence Pettit Bayesian Statistics - an Introduction

Plot for mixture prior

Example

I Suppose that we are about to introduce a new insurancepolicy. Interested in the mean number of claims per month.

I Prior?

I May expect that on average there would be 2 claims permonth, with a variance of 1.

I Conjugate prior for θ is Gamma(4,2):

p(θ) ∝ θ4−1 exp(−2θ)

Example

I Prior?

p(θ) ∝ θ4−1 exp(−2θ)

Example

I Prior?

p(θ) ∝ θ4−1 exp(−2θ)

Example

I Prior?

p(θ) ∝ θ4−1 exp(−2θ)

Prior density - Gamma(4,2)

Calculation of the posterior

I After 6 months there have been 18 claims.

I Assume that data follow a Poisson distribution with mean θ.

p(x |θ) =exp(−θ)θx

I The likelihood is given by

p(x |θ) ∝ θ∑

xi exp(−nθ)

where n = 6 and∑

xi = 18.I Posterior

p(θ|x) ∝ θ22−1 exp(−8θ)

which we can recognise is a Ga(22, 8).

I After 6 months there have been 18 claims.I Assume that data follow a Poisson distribution with mean θ.

p(x |θ) ∝ θ∑

xi exp(−nθ)

where n = 6 and∑

xi = 18.I Posterior

p(θ|x) ∝ θ22−1 exp(−8θ)

p(x |θ) ∝ θ∑

xi exp(−nθ)

where n = 6 and∑

xi = 18.

I Posteriorp(θ|x) ∝ θ22−1 exp(−8θ)

p(x |θ) ∝ θ∑

xi exp(−nθ)

where n = 6 and∑

xi = 18.I Posterior

p(θ|x) ∝ θ22−1 exp(−8θ)

I Posterior mean (228 ) is a weighted average of the prior mean

(2) and data mean (3).

I Weights are related to the prior and data variances.

I This holds for other common distributions.

Prior/Posterior density

Predictive distributions

I Distribution of a future observation y?

x = (x1, . . . , xn) is a random sample from p(xi |θ)

I Prior p(θ). Posterior p(θ|x).

I y - independent observation from the same distribution.Want p(y |x) - predictive distribution of y .

I y - independent observation from the same distribution.

Want p(y |x) - predictive distribution of y .

Predictive density

Now by extension of the argument

p(y |x) =

∫p(y |θ, x)p(θ|x)dθ

∫p(y |θ)p(θ|x)dθ

since y is independent of x given θ.

Poisson predictive example

I x1, . . . , xn ∼ Poisson(θ), prior for θ is Gamma(4,2)Posterior ⇒ Gamma(22,8)

I Suppose y ∼ Poisson(θ)

p(y |x) =

∫θy exp(−θ)

822θ21 exp−8θΓ(22)

Γ(22)y !

∫θ21+y exp−θ(9)dθ

Γ(22)y !

Γ(y + 22)

922+yy = 0, 1, 2, 3, . . .

Poisson predictive example

I x1, . . . , xn ∼ Poisson(θ), prior for θ is Gamma(4,2)Posterior ⇒ Gamma(22,8)

I Suppose y ∼ Poisson(θ)

p(y |x) =

∫θy exp(−θ)

822θ21 exp−8θΓ(22)

Γ(22)y !

∫θ21+y exp−θ(9)dθ

Γ(22)y !

Γ(y + 22)

922+yy = 0, 1, 2, 3, . . .

Predictive density of y

I The posterior distribution represents all our knowledge afterseeing the data.

I Why quote posterior mean and variance so much?

I Point estimates are justified by decision theory.

I As an extra ingredient we specify a loss function and choosean estimate to minimise expected loss.

I For example, quadratic loss

l(d , θ) = (d − θ)2

leads to the posterior mean as the estimator and posteriorvariance as the expected loss.

I Absolute error loss

l(d , θ) = |d − θ|

leads to the posterior median as the estimator.

I For example, quadratic loss

l(d , θ) = (d − θ)2

leads to the posterior mean as the estimator and posteriorvariance as the expected loss.

I Absolute error loss

l(d , θ) = |d − θ|

leads to the posterior median as the estimator.

Poisson hierarchical model

Stochastic Simulation - Gibbs sampling

Stochastic Simulation - basic idea:

I simulate from the distribution we are interested in

I use the simulated sample to make inferences.I Gibbs sampling is one of these methods.

Illustration of Gibbs

I Suppose we have three parameters θ, δ and φ.I Likelihood - p(x |θ, δ, φ).I Prior - p(θ, δ, φ).I Suppose we can find the conditional distributions:

p(θ|δ, φ, x), p(δ|φ, θ, x), p(φ|θ, δ, x)

that we can simulate from.

I simulate from the distribution we are interested inI use the simulated sample to make inferences.

I Gibbs sampling is one of these methods.

I simulate from the distribution we are interested inI use the simulated sample to make inferences.I Gibbs sampling is one of these methods.

I Suppose we have three parameters θ, δ and φ.

I Likelihood - p(x |θ, δ, φ).I Prior - p(θ, δ, φ).I Suppose we can find the conditional distributions:

I Suppose we have three parameters θ, δ and φ.I Likelihood - p(x |θ, δ, φ).

I Prior - p(θ, δ, φ).I Suppose we can find the conditional distributions:

I Suppose we have three parameters θ, δ and φ.I Likelihood - p(x |θ, δ, φ).I Prior - p(θ, δ, φ).

I Suppose we can find the conditional distributions:

that we can simulate from.Dr Lawrence Pettit Bayesian Statistics - an Introduction

Gibbs Sampling

I The Gibbs sampler is an iterative scheme:Step 1 Choose initial estimates θ0, δ0, φ0

Step 2 Given current estimates θi , δi , φi

simulate new valuesθi+1 from p(θ|δi , φi , x)δi+1 from p(δ|φi , θi+1, x)φi+1 from p(φ|θi+1, δi+1, x)

Step 3 Return to step 2

I The sequence (θ, δ, φ)i i = 1, 2, . . . is a realisation of aMarkov chain which, under mild regularity conditions, hasequilibrium distribution p(θ, δ, φ|x), the joint posteriordistribution of θ, δ, φ.

Gibbs Sampling

I The Gibbs sampler is an iterative scheme:Step 1 Choose initial estimates θ0, δ0, φ0

Step 2 Given current estimates θi , δi , φi

simulate new valuesθi+1 from p(θ|δi , φi , x)δi+1 from p(δ|φi , θi+1, x)φi+1 from p(φ|θi+1, δi+1, x)

Step 3 Return to step 2

I The sequence (θ, δ, φ)i i = 1, 2, . . . is a realisation of aMarkov chain which, under mild regularity conditions, hasequilibrium distribution p(θ, δ, φ|x), the joint posteriordistribution of θ, δ, φ.

Estimation from Gibbs Output

I Burn in period of length L.

I Posterior mean of θ can be estimated for large t by

L+t∑i=L+1

I To see a picture of the posterior density of θ - use Kerneldensity estimation.

I The predictive density of a new observation y can beestimated as

p(y |x) =1

L+t∑i=L+1

P(y |θi , δi , φi )

I Burn in period of length L.I Posterior mean of θ can be estimated for large t by

L+t∑i=L+1

p(y |x) =1

L+t∑i=L+1

p(y |x) =1

L+t∑i=L+1

p(y |x) =1

L+t∑i=L+1

I 1st stage - observations sj ∼ Poisson with mean tjλj forj = 1, 2, . . . , p, tj known

I 2nd stage - prior λj ∼ Ga(α, β) iid, α known

I 3rd stage - hyperprior β ∼ Ga(γ, δ), where γ and δ are known.

I p + 1 unknown parameters - λ’s and β.

I Joint posterior of all the parameters:

p∏j=1

(λj tj)sj exp(−λj tj)

sj !×

p∏j=1

Γ(α)−1βαλα−1j exp(−βλj)

×Γ(γ)−1δγβγ−1 exp(−δβ)

p∏j=1

sj !×

p∏j=1

sj !×

p∏j=1

sj !×

p∏j=1

sj !×

p∏j=1

Conditional distributions

I Conditional distributions of λ’s and β - pick out the relevantterms from this posterior.

p(λk |β, λj(j 6= k), data) ∝ λskk exp(−λktk)λα−1

k exp(−λkβ)

⇒ Ga(α + sk , β + tk)

p(β|λ, data) ∝ βpα exp(−∑

λjβ)βγ−1 exp(−δβ)

⇒ Ga(pα + γ,∑

λj + δ)

k exp(−λkβ)

⇒ Ga(α + sk , β + tk)

⇒ Ga(pα + γ,∑

λj + δ)

k exp(−λkβ)

⇒ Ga(α + sk , β + tk)

⇒ Ga(pα + γ,∑

λj + δ)

Gibbs runs

I For suitable starting values for the unknown parameters theGibbs sampler proceeds by drawing

λi+1j ∼ Ga(α + sj , β

i + tj) j = 1, . . . , p

βi+1 ∼ Ga(pα + γ,∑

λi+1j + δ)

I Use large sample of λ’s and β’s to estimate posterior andpredictive quantities of interest.

Gibbs runs

I For suitable starting values for the unknown parameters theGibbs sampler proceeds by drawing

λi+1j ∼ Ga(α + sj , β

i + tj) j = 1, . . . , p

βi+1 ∼ Ga(pα + γ,∑

λi+1j + δ)

I Use large sample of λ’s and β’s to estimate posterior andpredictive quantities of interest.

bayesian statistics - an introduction - qmul mathsfjw/goldsmiths/2008/lp/bayestalk.pdf · bayesian...

Documents