![Page 1: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/1.jpg)
Maximum Likelihood Estimation
Multivariate Normal distribution
![Page 2: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/2.jpg)
The Method of Maximum Likelihood
Suppose that the data x1, … , xn has joint density function
f(x1, … , xn ; 1, … , p)
where (1, … , p) are unknown parameters assumed to lie in (a subset of p-dimensional space).
We want to estimate the parameters1, … , p
![Page 3: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/3.jpg)
Definition: The Likelihood function Suppose that the data x1, … , xn has joint density function
f(x1, … , xn ; 1, … , p)
Then given the data the Likelihood function is defined to be
= L(1, … , p)
= f(x1, … , xn ; 1, … , p)
Note: the domain of L(1, … , p) is the set .
,f x
,f x L
![Page 4: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/4.jpg)
Definition: Maximum Likelihood Estimators
Suppose that the data x1, … , xn has joint density function
f(x1, … , xn ; 1, … , p)
Then the Likelihood function is defined to be
= L(1, … , p)
= f(x1, … , xn ; 1, … , p)
and the Maximum Likelihood estimators of the parameters 1, … , p are the values that maximize
= L(1, … , p)
,f x
L
L
![Page 5: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/5.jpg)
i.e. the Maximum Likelihood estimators of the parameters 1, … , p are the values
1
1 1, ,
ˆ ˆ, , max , ,p
p pL L
1̂ˆ, , p
Such that
Note: 1maximizing , , pL is equivalent to maximizing
1 1, , ln , ,p pl L
the log-likelihood function
![Page 6: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/6.jpg)
The Multivariate Normal Distribution
Maximum Likelihood Estiamtion
![Page 7: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/7.jpg)
Let 1 2, , nx x x
with mean vector
and covariance matrix
from the p-variate normal distribution
denote a sample (independent)
11 12 1
21 22 2
1 2
1 2
, , ,
n
n
n
p p pn
x x x
x x xx x x
x x x
Note:
![Page 8: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/8.jpg)
The matrix 1 2, , np n
x x xX
is called the data matrix.
11 12 1
21 22 2
1 2
n
n
p p pn
x x x
x x x
x x x
![Page 9: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/9.jpg)
The vector
1
2
1np
n
x
x
x
x
is called the data vector.
11
1
1
p
n
pn
x
x
x
x
![Page 10: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/10.jpg)
The mean vector
![Page 11: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/11.jpg)
The vector
1
2
1 2
1n
p
x
xx x x x
n
x
note
1 21
1 1 n
i i i in ijj
x x x x xn n
is called the sample mean vector
![Page 12: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/12.jpg)
also
1 11 12 1
2 21 22 2
1 2
1
11
1
n
n
p p p pn
x x x x
x x x xx
n
x x x x
11X
n
![Page 13: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/13.jpg)
In terms of the data vector
1
2
1
1 1, , ,
p npnp
n
x
xx I I I
n n
x
xA
where , , ,p np
I I IA
![Page 14: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/14.jpg)
Graphical representation of sample mean vector
2x
x1x
nx
2x
1x
px
The sample mean vector is the centroid of the data vectors.
![Page 15: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/15.jpg)
The Sample Covariance matrix
![Page 16: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/16.jpg)
The sample covariance matrix:
11 12 1
12 11 2
1 2
p
p
p p
p p pp
s s s
s s s
s s s
S
1
1
1
n
ik ij i kj kj
s x x x xn
where
![Page 17: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/17.jpg)
There are different ways of representing sample covariance matrix:
11 12 1
12 11 2
1 2
p
p
p p
p p pp
s s s
s s s
s s s
S
1 1 1
1
1
n
j jj p p
x x x xn
![Page 18: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/18.jpg)
1 1 1
1
1
n
j jj p p
S x x x xn
1
1
1,...,
1 n
n
x x
x x x xn
x x
1 1
1,..., ,..., ,..., ,...,
1 n nx x x x x x x xn
![Page 19: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/19.jpg)
1,..., ,..., ,..., ,...,
1 j j j jx x x x x x x xn
1 1 11,...,1 1,...,1
1X X X X
n n n
1 1 1
1X I J X I J
n n n
1 1
where 1,...,1 matrix of 1's
1 1n nJ n n
![Page 20: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/20.jpg)
1 1 1
1S X I J X I J
n n n
hence
1 1 1
1X I J I J X
n n n
1 1
1X I J X
n n
![Page 21: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/21.jpg)
Maximum Likelihood Estimation
Multivariate Normal distribution
![Page 22: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/22.jpg)
Let 1 2, , nx x x
with mean vector
and covariance matrix
from the p-variate normal distribution
denote a sample (independent)
11
21 / 2 1/ 2
1
1, , , e
2
i in x x
n pi
f x x
Then the joint density function of 1 2, , nx x x
is:
1
1
1
2
/ 2 / 2
1e
2
n
i ii
x x
np n
![Page 23: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/23.jpg)
The Likelihood function is:
1
1
1
2
/ 2 / 2
1, e
2
n
i ii
x x
np nL
and the Log-likelihood function is:
, ln , l L
1
1
1ln 2 ln
2 2 2
n
i ii
np nx x
![Page 24: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/24.jpg)
To find the Maximum Likelihood estimators of
1
1
1
2
/ 2 / 2
1, e
2
n
i ii
x x
np nL
or equivalently maximize
1
1
1, ln 2 ln
2 2 2
n
i ii
np nl x x
and
we need to find ˆ ˆ and
to maximize
![Page 25: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/25.jpg)
Note:
1
1
n
i ii
x x
thus 1
1
, 1
2
n
i ii
dl dx x
d d
1 1 1
1 1
2n n
i i ii i
x x x n
1 1
1
0n
ii
x n
1
1ˆn
ii
x xn
hence
![Page 26: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/26.jpg)
Now
1
1
1, ln 2 ln
2 2 2
n
i ii
np nl x x
1
1
1ln 2 ln tr
2 2 2
n
i ii
np nx x
1
1
1ln 2 ln tr
2 2 2
n
i ii
np nx x
tr tr AB BA
1
1
1ln 2 ln tr
2 2 2
n
i ii
np nx x
![Page 27: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/27.jpg)
Now ,l
1
1
, ln 1tr
2 2
n
i ii
dl dn dx x
d d d
1
1
1ln 2 ln tr
2 2 2
n
i ii
np nx x
1 1 1
1
10
2 2
n
i ip p
i
nx x
1
1 ˆ ˆˆor n
i ii
x xn
1
1 1=
n
i ii
nx x x x S
n n
![Page 28: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/28.jpg)
and
Summary:
the Maximum Likelihood estimators of
are
1
1ˆ n
ii
x xn
and
1
1 1ˆ n
i ii
nx x x x S
n n
![Page 29: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/29.jpg)
Sampling distribution of the MLE’s
![Page 30: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/30.jpg)
Note
1
1
1 1ˆ , ,n
ii
n
x
x x I I Axn n
x
11
21 / 2 1/ 2
1
1, , , e
2
i in x x
n pi
f x x
The joint density function of 1 2, , nx x x
is:
1
1
1
2
/ 2 / 2
1e
2
n
i ii
x x
np n
![Page 31: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/31.jpg)
*
0
and covariance matrix
0
p p
p p
This distribution is np-variate normal with mean vector
*
![Page 32: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/32.jpg)
1
1
1 1ˆ , ,n
ii
n
x
x x I I Axn n
x
Thus the distribution of
is p-variate normal with mean vector
*
1
1 1 1, , = =
n
i
A I I nn n n
![Page 33: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/33.jpg)
*and covariance matrix A A
2
2
01
, ,
0
1 1=
p p
p p
I
I In
I
nn n
![Page 34: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/34.jpg)
Summary
The sampling distribution of
is p-variate normal with
x
nxx
1 and
![Page 35: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/35.jpg)
The sampling distribution of the sample covariance matrix S
and
Sn
n 1ˆ
![Page 36: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/36.jpg)
The Wishart distribution
A multivariate generalization of the 2 distribution
![Page 37: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/37.jpg)
Let 1 2, , , kz z z be k independent random p-vectors
Each having a p-variate normal distribution with
1mean vector 0 and covariance matrix
p pp
and covariance matrix p p
1 1 2 2Let k kp pU z z z z z z
Then U is said to have the p-variate Wishart distribution with k degrees of freedom
pU W k
Definition: the p-variate Wishart distribution
![Page 38: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/38.jpg)
Suppose
Then the joint density of U is:
1 / 4
1
i.e. / 2 1 / 2p
p pp
j
k k j
1 / 2 12
/ 2/ 2
exp
2 / 2
k p
U kkpp pp
u tr uf u
k
where p(·) is the multivariate gamma function.
pU W k
The density ot the p-variate Wishart distribution
It can be easily checked that when p = 1 and 1 then the Wishart distribution becomes the 2
distribution with k degrees of freedom.
U
![Page 39: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/39.jpg)
Suppose
Let denote a matrix of rank .q pC q p q p
pU W k
Theorem
then
2 21 a kv a Ua W k a a
Corollary 1:
2with a a a
Corollary 2: If the diagonal element of th
iiu i U 2then where ii ii k iju
pV CUC W k C C
Proof
Set [0 0 1 0]i
i
a e
![Page 40: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/40.jpg)
Suppose 1 1 2 2 and p pU W k U W k
Theorem
are independent, then
1 2 1 2pV U U W k k
Suppose 1 1 2 and pU W k UTheorem
are independent and
1 2 1 with pV U U W k k k
then 2 1pU W k k
![Page 41: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/41.jpg)
1
n
i i pi
U x x W n
Theorem Let
Theorem
1 2, , , nx x x
be a sample from
then pN
1 1i pU n x x W
Let 1 2, , , nx x x
be a sample from
then pN
![Page 42: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/42.jpg)
1
n
i ii
U x x
Theorem
Proof
in x x
1
n
i ii
x x x x
1
n
i ii
U x x
1
n
i ii
x x x x x x
etc
![Page 43: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/43.jpg)
21
n
i ii
U x x x x
Theorem Let
1 2, , , nx x x
be a sample from
then
pN
1 iU n x x
is independent of
Proof 1 1 1
21 22 2
1 2
Let
n n n
n
n n nn
h h hH
h h h
be orthogonal
Then H H HH I
![Page 44: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/44.jpg)
1 1 1
* 21 22 2
1 2
Let
n n n
n
np np
n n nn
I I I
h I h I h IH
h I h I h I
Note H* is also orthogonal* the Kronecker product of and H H I H I
11 1
1
n
m mn
a B a B
A B
a B a B
![Page 45: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/45.jpg)
Properties of Kronecker-product
1. A B C D AC BD
2. A B A B
1 1 1 3. A B A B
BDACDCBA
![Page 46: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/46.jpg)
1 1 11 1
2 2* 21 22 2
1 2
Let
n n n
n
n nn n nn
I I I x u
x uh I h I h IH x
x uh I h I h I
11
1
1
for 2,3,...,
n
ini
n
i ij ji
u x nx
u h x i p
![Page 47: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/47.jpg)
1 1
Note: n n
i i i ii i
u u x x
1 12 1
n n
i i i ii i
u u u u x x
1 1 1 12 1 1
- - 1 n n n
i i i i i ii i i
u u u u x x u u x x nxx n S
![Page 48: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/48.jpg)
*
0
and covariance matrix =
0
p p
p p
I
This the distribution of
* 1
is np-variate normal with mean vector 1 2, , , nx x x x
![Page 49: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/49.jpg)
= H I I H I
Thus the joint distribution of
1
= 1 =
0
n
H
�
is np-variate normal with mean vector u H I x
1 u H I H I
*and covariance matrix u H I H I
= HH I
![Page 50: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/50.jpg)
0
and covariance matrix =
0
p p
u
p p
I
Thus the joint distribution of
1
=
0
n
u
is np-variate normal with mean vector
u
![Page 51: Maximum Likelihood Estimation Multivariate Normal distribution](https://reader033.vdocuments.us/reader033/viewer/2022061607/56649dba5503460f94aab3a3/html5/thumbnails/51.jpg)
1
1 1n
i i pi
U x x x x n S W n
Summary: Sampling distribution of MLE’s for multivatiate Normal distribution
Let 1 2, , , nx x x
be a sample from
then
pN
1p nx N
and
22 2
1 1Also 1ii ii
nu s n