bayesian hierarchical model ying nian wu ucla department of statistics ipam summer school
DESCRIPTION
Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School July 12, 2007. Plan Bayesian inference Learning the prior Examples Josh’s example. Inference of normal mean. independently. unknown parameter. given constant. Example:. one’s height. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/1.jpg)
Bayesian Hierarchical Model
Ying Nian WuUCLA Department of Statistics
IPAM Summer SchoolJuly 12, 2007
![Page 2: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/2.jpg)
Plan
•Bayesian inference•Learning the prior•Examples•Josh’s example
![Page 3: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/3.jpg)
independently
2
unknown parameter
given constant
one’s height
Inference of normal mean
Example:
),(~]|,...,,[ 221 NnYYY
nYYY ,...,, 21repeated measurements
2 known precision
![Page 4: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/4.jpg)
Prior distribution
),(~ 2 N
),( 2 known hyper-parameters
The larger 2 , the more uncertain about 2 , prior becomes non-informative
![Page 5: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/5.jpg)
Bayesian inference
independently),(~]|,...,,[ 221 NnYYY
Prior: ),(~ 2 N
Data:
Posterior: )1
1,
1
1
(~],...,|[
2222
22
1
nn
nY
YY n N
n
jjYn
Y1
1
Compromise between prior and data
![Page 6: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/6.jpg)
Bayesian inference
Prior:
Data:
)(~ p
)|(~]|[ ypY
Posterior: )|()()|(~]|[ yppypyY
likelihoodpriorposterior
![Page 7: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/7.jpg)
Prior:
Data:
)(~ p
)|(~]|[ ypY
y
)|()(),(~],[ yppypY
]|[ yY
]|[ Y
Illustration
![Page 8: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/8.jpg)
independently),(~]|,...,,[ 221 NnYYY
Prior: ),(~ 2 N
Data:
Inference of normal mean
Sufficient statistic: ),(~1
]|[2
1 nY
nY
n
jj
N
Y
]|[ Y
]|[ Y
![Page 9: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/9.jpg)
Combining prior and data
Y
]|[ Y
2 large n/2 small
![Page 10: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/10.jpg)
Combining prior and data
]|[ YY
2 largen/2small
![Page 11: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/11.jpg)
Y
]|[ Y
)1
1,
1
1
(~],...,|[
2222
22
1
nn
nY
YY n N
Prior knowledge is useful for inferring
![Page 12: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/12.jpg)
independently),(~]|,...,,[ 221 NnYYY
Prior: ),(~ 2 N
Data:
Learning the prior
Prior distribution cannot be learned from single realization of
![Page 13: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/13.jpg)
Prior:
Data:
Learning the prior
mii ,...,1),,(~],|[ 22 N
iiiij njY ,...,1),,(~],,|[ 22 N
Prior distribution can be learned from multiple experiences
![Page 14: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/14.jpg)
Prior:
Data:
mii ,...,1),,(~],|[ 22 N
iiiij njY ,...,1),,(~],,|[ 22 N
Hierarchical model
),( 2
1 2 i m…… ……
1,12,11,1 ,...,, nYYYiniii YYY ,2,1, ,...,,
mnmmm YYY ,2,1, ,...,,
![Page 15: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/15.jpg)
1 2 i m…… ……
1Y iY mY2Y
Hierarchical model
![Page 16: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/16.jpg)
Collapsing
iiiii dppp )|()|()|( YY
yprojecting
![Page 17: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/17.jpg)
Prior:
Data:
mii ,...,1),,(~],|[ 22 N
iiiij njY ,...,1),,(~],,|[ 22 N
Sufficient statistics
in
jij
ii Yn
Y1
1
),(~]|[2
iiii n
Y N
![Page 18: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/18.jpg)
),(~],|[2
22
ii nY
N
),(~]|[2
iiii n
Y N
Integrating out i
Collapsing
![Page 19: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/19.jpg)
),(~],|[2
22
nYi
N
Estimating hyper-parameter
m
iiYm 1
1
nY
m
m
ii
2
1
22 )ˆ(1
ˆ
![Page 20: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/20.jpg)
Empirical Bayes
Borrowing strength from other observations
22
22
ˆ1
ˆ1
ˆˆ
n
nYi
i
iY
i
![Page 21: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/21.jpg)
Hyper prior: )(~),( 2 p e.g., constant
),( 2
1 2 i m…… ……
1,12,11,1 ,...,, nYYYiniii YYY ,2,1, ,...,,
mnmmm YYY ,2,1, ,...,,
Full Bayesian
![Page 22: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/22.jpg)
Full Bayesian
)]|()|([)(~],...,;,...,;[1
11 ii
m
iimm ppp YYY
)]|()|([)(),...,|,...,;(1
11 ii
m
iimm pppp YYY
![Page 23: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/23.jpg)
m
iim ppp
11 )|()(),...,|( YYY
m
iiimmm dppp
1111 ),|(),...,|(),...,|,...,( YYYYY
Bayesian hierarchical model
![Page 24: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/24.jpg)
Stein’s estimator
miY iii ,...,1),,(~]|[ 2 N
Example: measure each person’s height
ii Y 22
1
)ˆ( mm
iii
E
![Page 25: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/25.jpg)
im
ii
i YY
m)
)2(1(
~
1
2
2
2
1
2)~
( mm
iii
E3m
Stein’s estimator
![Page 26: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/26.jpg)
Stein’s estimator
222 ][ iiYE222 ][ mY
ii
ii E
miY iii ,...,1),,(~]|[ 2 N
Y
![Page 27: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/27.jpg)
miY iii ,...,1),,(~]|[ 2 N
Stein’s estimator
),0(~ 2 Ni
Empirical Bayes interpretation
![Page 28: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/28.jpg)
Beta-Binomial example
),(~]|[ nY Binomial
e.g., flip a coin, is probability of head
Y is number of heads out of n flips
yny
y
nyYp
)1()|(
Data:
Pre-election poll
![Page 29: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/29.jpg)
),(~ 01 aaBeta
11
01
01 01 )1()()(
)()(
aa
aa
aap
01
1][aa
a
E
Conjugate prior
![Page 30: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/30.jpg)
),(~]|[ nY Binomial
yny
y
nyYp
)1()|(
Data:
11
01
01 01 )1()()(
)()(
aa
aa
aap
),(~ 01 aaBetaPrior:
Posterior: ),(~]|[ 01 aynayyY Beta
01
1~]|[aan
ayyY
E
![Page 31: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/31.jpg)
Hierarchical model
Examples: a number of coins probs of head a number of MLB players probs of hit pre-election poll in different states
),(~ 01 aai Beta
![Page 32: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/32.jpg)
Dirichlet-Multinomial
Roll a die: ),...( 61
),(~]|,...,[ 61 nYY lmultinomia
6161
6161 ...
,...,)|,...,( yy
yy
nyyp
![Page 33: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/33.jpg)
Conjugate prior
6161
6161 ...
,...,)|,...,( yy
yy
nyyp
16
11
61
61 61 ...)()...(
)...()(
aa
aa
aap
),...,(~ 61 aaDirichlet
![Page 34: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/34.jpg)
),(~]|,...,[ 61 nYY lmultinomia
),...,(~ 61 aaDirichlet
),...,(~]|[ 6611 ayayy Dirichlet
Data
Prior
Posterior
61 ...~]|[
aan
ayy kk
k
E
![Page 35: Bayesian Hierarchical Model Ying Nian Wu UCLA Department of Statistics IPAM Summer School](https://reader036.vdocuments.us/reader036/viewer/2022062517/56813ae3550346895da33c22/html5/thumbnails/35.jpg)
Hierarchical model
),...,(~ 61 aai Dirichlet