the wishart and inverse-wishart distribution
DESCRIPTION
The Wishart and inverse-wishart distributionTRANSCRIPT
![Page 1: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/1.jpg)
Pankaj Das
M.Sc. (Agricultural Statistics)
Roll no. – 20394
![Page 2: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/2.jpg)
Introduction
Mathematical background
Wishart distribution
Inverse- Wishart distribution.
Relationship of Wishart and Inverse-Wishart distribution
Conclusion
![Page 3: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/3.jpg)
In the modern Era of science and information technology, there has
been a huge influx of high-dimensional data from various fields
such as genomics, environmental sciences, finance and the social
sciences.
In sense of all the many complex relationships and multivariate
dependencies present in the data and formulating correct models and
developing inferential procedures are the major challenges in
modern day statistics.
![Page 4: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/4.jpg)
In parametric models the covariance or correlation matrix (or its
inverse) is the fundamental object that quantifies relationships
between random variables.
Estimating the covariance matrix in a sparse way is crucial in high
dimensional problems and enables the detection of the most
important relationship.
Covariance matrices provide the simplest measure of dependency,
and therefore, much attention has been placed on modelling
covariance matrices. It has a significant impact on statistical
inferences.
![Page 5: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/5.jpg)
In short correlation matrix plays vital role in multivariate statistics.
The correlation/covariance matrix is directly involved in a variety of
statistical models. Estimation of correlation matrix is important.
In the estimation of covariance matrices in multivariate statistics
Wishart & inverse wishart distribution is used.
![Page 6: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/6.jpg)
Suppose is p-random variable drawn from a p-variate normal
distribution with mean vector and covariance matrix ∑.
Then the joint density function of is given by
P α
T
α ααN ( ,= , ..., ~ ) α = 1, ..., ; n X x x μ Σ
...(1)
(α)X
1 2, , nx x x
1 -1- x -μ Σ x -μn i i1 2f x ,…,x μ,Σ = e 1 n p/2 1/2i=1 2π Σ
n1 -1- x -μ Σ x -μi i21 i=1 = enp/2 n/2
2π Σ
μ
![Page 7: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/7.jpg)
x1
x x xx 11 12 1n2x x x. 21 22 2n= =p×n
.
x x. xpnp1 p2xp
X
p×nX Where is called data matrix which is given by
The Mean vector is shown as
p×1
n
α
α=1
1x= x
n
x1
x2
1 .= x +x …+x =1 n2
n .
.
xp
![Page 8: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/8.jpg)
The variance-covariance matrix is given by
Another way
This dispersion matrix is semi-positive definite.
11 1
1
ˆn
p pn
'
1
1ˆ (x )( )1
n
x x xn
![Page 9: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/9.jpg)
The sample covariance matrix
Where,
11 12 1
12 11 2
1 1 1
1 2
1
1
p
np
j jp p j p p
p p pp
s s s
s s sx x x x
n
s s s
s
1
1
1
n
ik ij i kj k
j
s x x x xn
![Page 10: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/10.jpg)
The sampling distribution of the sample covariance matrix S and
follow a distribution which is known as Wishart
distribution.
It is named in honor of John Wishart, who first formulated the
distribution in 1928.
n-1Σ̂= S
n
j
n2(n-1)s = (x -x)(x -x)j=1
j p~W n-1, Σ
![Page 11: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/11.jpg)
The Wishart distribution is a multivariate extension of the Gamma
distribution. It simplifies to a multivariate generalization of the χ2
distribution.
χ2 distribution describes the sum of squares of n draws from a
univariate normal distribution, where the Wishart distribution
represents the sum of squares (and cross-products) of n draws from a
multivariate normal distribution.
It is a family of probability distributions defined over symmetric,
nonnegative-definite matrix-valued random variables.
![Page 12: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/12.jpg)
The Wishart distribution arises as the distribution of the sample
covariance matrix for a sample from a multivariate normal
distribution.
Contd...
![Page 13: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/13.jpg)
Let be k independent random p-vectors . Each having a
p-variate normal distribution with mean vector and covariance
matrix
Let
Then U is said to have the p-variate Wishart distribution with k
degrees of freedom and the covariance matrix , where .
1 2 kz ,z ,…,z
1 0 p
p p
' ' '
1 1 2 2 k kp×p
=z z +z z +…+z zU
~ pU W k
p×10
p p
![Page 14: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/14.jpg)
The joint density of the p-variate Wishart distribution is
where is a multivariate gamma function.
i.e.
This is known as central Wishart distribution.
1 /212
/2/2
exp
2 / 2
k p
U kkpp pp
u tr uf u
k
1 /4
1
/ 2 1 / 2p
p p
p
j
k k j
...(2)
...(3)(.)
Contd...
(.)
![Page 15: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/15.jpg)
Let
Then where M is a matrix with columns
Where &
This is called non-central Wishart distribution.
, . .., ~ N ( , ) ; 1,2,...,kT
pX x x
'
1
~ ( , ,M)k
px x W k
p k 1 2, ,..., k
11 1
'
1
k
p pk
x x
x x
11 1
1
' ( ')
k
p pk
M
![Page 16: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/16.jpg)
When M = 0 then Wp is called central wishart distribution & we
write as
It can be easily checked that when p = 1 and ∑= 1 then the Wishart
distribution becomes the chi-square distribution with k degrees of
freedom. Note that we must have k> p-1 to ensure ∑ is invertible. If
k > p-1 does not hold, is called singular Wishart distribution due to
∑ being a singular matrix.
(k, )pW
![Page 17: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/17.jpg)
The expected value of S is
...(4)
The expected value of a Wishart distribution depends on number of
draws one makes from the multivariate normal distribution.
In comparison, the expected value of a distribution is k, so that
the only differences between a Wishart expectation and a
expectation are the underlying dimensionality of the data and a
scale component.
(S) k
2 ( )k
2 ( )k
~ W(k , )i iS Let 1 ,...,i n where
![Page 18: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/18.jpg)
We can find the individual variances of the elements of S. for instance,the variance of the ijth element of S is:
...(5)
Where is the ijth element of the matrix and can be thought of as the
population covariance between variable i and variable j. When p = 1, so
that the only element of the variance/covariance matrix is
Therefore, we get Var(X) = k (1 + 1 x 1) = 2k, which is the familiar
variance of a variable.
2 ij ij ii j jVar S k
ij thij
2 k
thij
2
11 11 1
ij
2 ( )k
![Page 19: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/19.jpg)
Equation (5) is a set of variances rather than depicting the
variance/covariance matrix because every observation of the Wishart
distribution is a matrix.
Therefore, describing all combinations of variances and covariance
of S requires either an array of higher order or a Kronecker
operation to represent that higher order array as a matrix
![Page 20: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/20.jpg)
In 2007 Eaton et al. present a new idea to derive for derivation the
covariance matrix of S. The covariance matrix of S can be represented
as
Where is the Cholesky decomposition of the (square,symmetric) matrix , and
1
Cov(S) cov( )k
T
i i
i
x x
1
( )k
T
i i
i
Cov x x
(Cz C )T T
i ikCov z
TCC
T
i i pE z z I
...(6)
![Page 21: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/21.jpg)
Applying the vector(vec) operator to S, which forms a long vector
by stacking the columns of S, so that Cov[vec(s)] is a matrix rather
than an array, so we have
k T TCov vec S Cov vec Czz C
TkCov C C vec zz
TTk C C Cov vec zz C C
T Tk C C Cov z z C C
(by the vec to Kronecker property)
(by vec and Kronecker properties)
z
![Page 22: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/22.jpg)
To determine cov [vec(s)] (as a proxy for cov(s)), one would only
need to know
(1) The variance of (where n is any element of z);
(2) The variance of (where are any two elements of z);
(3) The covariance between & ;
(4) The covariance between & and
(5) The covariance between and (where at most two of i, j, n, or
o are the same).
Cov z z
n oZ Z n o
n oZ Zo nZ Z
n oZ Z
2
nZ 2
oZ
2
nZ
![Page 23: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/23.jpg)
1. is standard normally distributed, so follows a distribution
variance equal to 2(1) = 2. Therefore, = 2 for all k.
2. & are uncorrelated standard normal random variables, which
implies that they are also independent. Therefore,
due to independence
Here & both follow a distribution.
nZ
2 )(1
2 2 2 2 2 0n o n o n oVar Z Z E Z Z E Z E Z E Z Z E Z E Z
2 2( ) ( )( ) 1 1 1n o n oVar Z Z E Z Z
2 )(1
nZ
2( )nV Z
![Page 24: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/24.jpg)
2 22 2— — 1— 0 1n o o n n o o n n o o n n o n oCov Z Z Z Z E Z Z Z Z E Z Z E Z Z E Z Z E Z E Z
3. & uncorrelated standard normal random variables,
so
4. 2 2 2 2 2 2( 1 1 0, ) —n o n o n oCov Z Z E Z Z E Z E Z
5. , 0i j n oCov Z Z Z Z
![Page 25: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/25.jpg)
Therefore, the [p(n-1)+ n,p(n -1)+n] elements of will all be 2
because and the remaining diagonal elements of
will all be 1 because for all k = l. The off-diagonal elements
must be 0 except for those elements symbolizing the covariance
between and , which will be 1.
Ultimately can be written as
is a matrix of 1s and 0s.
2 2kVar Z
1n oVar Z Z
p p pI I M
2 2 p ppM
i jZ Z j iZ Z
pM 2 2 p p
i jZ Z j iZ Z
![Page 26: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/26.jpg)
Therefore,
T TCov vec S k C C Cov z z C C
T T
p p pk C C I I M C C
T T T T
pk C C C C C C M C C
T T
pk C C M C C ...(7)
![Page 27: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/27.jpg)
And we can check the derivation by simulating draws from a
Wishart distribution and comparing the simulated covariance with
the empirical covariance matrix calculated using equation (7).
Analysis is done with the help of R console.
![Page 28: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/28.jpg)
Results shows that replications must be very large to be for theempirical covariance matrix to be close to the theoretical covariancematrix.
![Page 29: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/29.jpg)
Some Theorems of Wishart Distribution
![Page 30: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/30.jpg)
Some Theorems on Wishart Distribution
![Page 31: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/31.jpg)
Wishart distribution have great importance in the estimation of
covariance matrices in multivariate statistics.
The Wishart distribution is frequently used as the prior on the
precision matrix parameter of a multivariate normal distribution.
The Wishart distribution arises as models for random variation and
descriptions of uncertainty about variance and precision matrices. They
are of particular interest in sampling and inference on covariance and
association structure in multivariate normal models, and in ranges of
extensions in regression.
![Page 32: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/32.jpg)
The Inverse-Wishart distribution is the multivariate extension of the
Inverse-Gamma distribution.
Even though the Wishart distribution generates sums of squares
matrices, one can think of the Inverse-Wishart distribution as
generating random covariance matrices.
However, those covariance matrices would be inverses of the
covariance matrices generated under Wishart distribution.
![Page 33: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/33.jpg)
Let T ~ InvWishp
Where denotes a positive definite scale matrix, m denotes the degrees of
freedom, and p indicates the dimensions of T (i.e. ).
Then T is positive definite with probability density function is given as
...(8)
m2| Ψ | 1 -1= exp[- tr(ΨT )]
m+p+1 mp 2m2 2| T | 2 Γ ( )p
f
2
T =
( , )m
Ψp×pT R
![Page 34: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/34.jpg)
The expected value of T is
...(9)
In respect to the distribution, the only differences between the
Inverse-Wishart expectation and the inverse- expectation are the
dimensionality of the data and a scale component.
The Inverse-Wishart distribution has finite expectation only when m >
(p -1).
( )1
Tm p
2χ
![Page 35: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/35.jpg)
The variance of the ijth element of T is
Where is the element of the matrix
If p = 1, then only element of the variance/covariance matrix is given by
The variance expression is given by
Which is same as the variance of variable.
2
2
( 1) ( 1)(T )
( )( 1) ( 3)
ij ii jj
ij
m p m pVar
m p m p m p
2
11 11 1
2 2 2
( ) 1 ( 2) 1 1 2( 1) 2( )
( 1)( 2) ( 4) ( 1)( 2) ( 4) ( 2) ( 4)
m m mVar X
m m m m m m m m
2Inv-χ (m)
ijψ
![Page 36: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/36.jpg)
k ~ ( , )pS W
1 1~ ),(pInvWish mS
The Wishart distribution is related to the normal distribution, Chi-square
distribution and Gamma distribution . The Inverse-Wishart distribution
is related to the those distributions in a similar way.
let
then
Where m=k is the degrees of freedom
![Page 37: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/37.jpg)
![Page 38: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/38.jpg)
![Page 39: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/39.jpg)
The Inverse-Wishart distribution is frequently used as the prior on
the variance/covariance matrix parameter (S) of a multivariate
normal distribution. Note that the Inverse-Gamma distribution is the
conjugate prior for the variance parameter of a univariate normal
distribution, and the Inverse-Wishart distribution (as its multivariate
generalization) extends conjugacy to the multivariate normal
distribution.
![Page 40: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/40.jpg)
The Wishart and Inverse-Wishart distribution is an important
distribution having a certain good and useful statistical properties.
These distributions have important role in estimating parameter in
multivariate studies.
Wishart distribution help to develop a framework for bayesian
inference for Gaussian covariance graph models.
Hence this work completes the powerful theory that has been developed
in the mathematical statistics literature for decomposable models. This
models help to draw a valid inference.
![Page 41: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/41.jpg)
The generalized Wishart process (GWP) – which we used to model
time-varying covariance matrices σ(t). In the future, the GWP could be
applied to study how Σ depends on covariates like interest rates.
Conclusion:
![Page 42: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/42.jpg)
Anderson, T. W.(2003). An Introduction to Multivariate StatisticalAnalysis. Wiley India (P) Ltd., New Delhi.
Chatfield, C. and Collins, A. J.(1980). Introduction to Multivariate
Analysis. Chapman and Hall-CRC, London.
Chib, S. and Greenberg, E.(1998). Bayesian analysis of multivariate
probit models. Biometrika , 85, 347-361.
Cook, R. D.(2011). On the mean and variance of the generalized
inverse of a singular Wishart matrix . Electronic Journal of Statistics
, 5, 146-158.
![Page 43: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/43.jpg)
Eaton, M. L.(2007). Multivariate statistics: A vector space approach , .
Wiley India (P) Ltd., New Delhi.
Nydick, S. W.(2012). The Wishart and inverse Wishart distributions.
International Journal of Electronics and Communication,
22, 119-139.
Pourahmadi, M., Daniels, M. J. and Park, T.(2006). Simultaneously
modelling of the cholesky decomposition of several covariance
matrices. Journal of Multivariate Analysis, 97, 125-135.
![Page 44: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/44.jpg)
Rao, C. R.(1965). Linear statistical inference and its applications,Wiley India (P) Ltd., New Delhi.
![Page 45: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/45.jpg)
![Page 46: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/46.jpg)
It is a decomposition of hermitian positive definitematrix into the product of lower triangular matrixand it conjugate transpose.
The cholesky decomposition of a hermitian positive-definite matrix a is a decomposition of the form.
Where L is a lower triangular matrix with real andpositive diagonal entries, and L* denotes theconjugate transpose of L
A LL
Cholesky decomposition
![Page 47: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/47.jpg)
R CODE
![Page 48: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/48.jpg)
M
![Page 49: The Wishart and inverse-wishart distribution](https://reader034.vdocuments.us/reader034/viewer/2022052304/5597dfbd1a28ab52388b47ef/html5/thumbnails/49.jpg)
Any
Question
s ????