bayes factor
DESCRIPTION
Bayes Factor. Based on Han and Carlin (2001, JASA). Model Selection. Data: y : finite set of competing models : a distinct unknown parameter vector of dimension n j corresponding to the j th model Prior : all possible values for - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/1.jpg)
Bayes Factor
Based on Han and Carlin (2001, JASA)
![Page 2: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/2.jpg)
Model Selection
• Data: y• : finite set of competing models• : a distinct unknown parameter vector of
dimension nj corresponding to the jth model• Prior• : all possible values for
• : collection of all model specific
![Page 3: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/3.jpg)
Model Selection
• Posterior probability
– A single “best” model– Model averaging
• Bayes factor: Choice between two models
![Page 4: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/4.jpg)
Estimating Marginal Likelihood
• Marginal likelihood
• Estimation– Ordinary Monte Carlo sampling
• Difficult to implement for high-dimensional models• MCMC does not provide estimate of marginal likelihood
directly
– Include model indicator as a parameter in sampling• Product space search by Gibbs sampling• Metropolis-Hastings• Reversible Jump MCMC
![Page 5: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/5.jpg)
Product space search
• Carlin and Chib (1995, JRSSB)• Data likelihood of Model j• Prior of model j• Assumption:
– M is merely an indicator of which is relevant to y– Y is independent of given the model
indicator M– Proper priors are required– Prior independence among given M
![Page 6: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/6.jpg)
Product space search• The sampler operates over the product space
• Marginal likelihood
• Remark:
![Page 7: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/7.jpg)
Search by Gibbs sampler
![Page 8: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/8.jpg)
Bayes Factor
• Provided the sampling chain for the model indicator mixes well, the posterior probability of model j can be estimated by
• Bayes factor is estimated by
![Page 9: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/9.jpg)
Choice of prior probability
• In general, can be chosen arbitrarily– Its effect is divided out in the estimate of Bayes
factor• Often, they are chosen so that the algorithm
visits each model in roughly equal proportion– Allows more accurate estimate of Bayes factor– Preliminary runs are needed to select
computationally efficient values
![Page 10: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/10.jpg)
More remarks
• Performance of this method is optimized when the pseudo-priors match the corresponding model specific priors as nearly as possible
• Draw back of the method– Draw must be made from each pseudo prior at
each iteration to produce acceptably accurate results
– If a large number of models are considered, the method becomes impractical
![Page 11: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/11.jpg)
Metropolized product space search
• Dellaportas P., Forster J.J., Ntzoufras I. (2002). On Bayesian Model and Variable Selection Using MCMC. Statistics and Computing, 12, 27-36.
• A hybrid Gibbs-Metropolis strategy• Model selection step is based on a
proposal moving between models
![Page 12: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/12.jpg)
Metropolized Carlin and Chib (MCC)
![Page 13: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/13.jpg)
Advantage
• Only needs to sample from the pseudo prior for the proposed model
![Page 14: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/14.jpg)
Reversible jump MCMC
• Green (1995, Biometrika)• This method operates on the union space
• It generates a Markov chain that can jump between models with parameter spaces of different dimensions
![Page 15: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/15.jpg)
RJMCMC
![Page 16: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/16.jpg)
Using Partial Analytic Structure (PAS)
• Godsill (2001, JCGS)• Similar setup as in the CC method, but allows
parameters to be shared between different models.
• Avoids dimension matching
![Page 17: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/17.jpg)
PAS
![Page 18: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/18.jpg)
Marginal likelihood estimation (Chib 1995)
![Page 19: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/19.jpg)
Marginal likelihood estimation (Chib 1995)
• Let When all full conditional distributions for the parameters are in closed form
![Page 20: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/20.jpg)
Chib (1995)
• The first three terms on the right side are available in close form
• The last term on the right side can be estimated from Gibbs steps
![Page 21: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/21.jpg)
Chib and Jeliazkov (2001)
• Estimation of the last term requires knowing the normalizing constant– Not applicable to Metropolis-Hastings
• Let the acceptance probability be
![Page 22: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/22.jpg)
Chib and Jeliazkov (2001)
![Page 23: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/23.jpg)
Example: linear regression model
![Page 24: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/24.jpg)
Two models
![Page 25: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/25.jpg)
Priors
![Page 26: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/26.jpg)
For MCC
• h(1,1)=h(1,2)=h(2,1)=h(2,2)=0.5
![Page 27: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/27.jpg)
For RJMCMC
• Dimension matching is automatically satisfied• Due to similarities between two models
![Page 28: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/28.jpg)
Chib’s method
• Two block gibbs sampler– (regression coefficients, variance)
![Page 29: Bayes Factor](https://reader035.vdocuments.us/reader035/viewer/2022062410/56815df5550346895dcc2c06/html5/thumbnails/29.jpg)
Results• By numerical integration, the true Bayes factor
should be 4862 in favor of Model 2