1 quadratic forms, characteristic roots and characteristic vectors mohammed nasser professor, dept....
TRANSCRIPT
![Page 1: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/1.jpg)
1
Quadratic Forms, Characteristic Roots and Characteristic Vectors
Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh
Email: [email protected]
1
The use of matrix theory is now widespread .- - - -- are essential in ----------modern treatment of univeriate and multivariate statistical methods. ----------C.R.Rao
![Page 2: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/2.jpg)
2
Contents
Linear Map and Matrices
Quadratic Forms and Its Applications in MM
Classification of Quadratic Forms
Quadratic Forms and Inner Product
Definitions of Characteristic Roots and Characteristic Vectors
Geometric Interpretations
Properties of Grammian Matrices
Spectral Decomposition and Applications
Matrix Inequalities and Maximization
Computations
2
![Page 3: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/3.jpg)
3
Statistical Concepts/Techniques
Concepts in Vector space
Variance Length of a vector, Qd. forms
Covariance Dot product of two vectors
Correlation Angle bt.two vectors
Regression and Classification
Mapping bt two vector sp.
PCA/LDA/CCA Orthogonal/oblique projection on lower dim.
Relation between MM (ML) and Vector space
![Page 4: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/4.jpg)
4
Some Vector Concepts
• Dot product = scalar
ii
iT yxyxyxyx
y
y
y
xxx
3
1332211
3
2
1
321yx
|| x || = (x12+ x2
2 + x32 )1/2
Inner product of a vector with itself = (vector length)2
xT x =x12+ x2
2 +x32 = (|| x
||)2
x1
x2 ||x||
Right-angle triangle
Pythagoras’ theorem
• Length of a vector
1
21 2
1
...n
Tn i i
i
n
y
yx x x xy
y
x y
|| x || = (x1
2+ x22)1/2
x1
x2
![Page 5: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/5.jpg)
5
2 1
2 1
1 1 2 2
sin cos
sin cos
cos cos( ) cos cos sin sin
x ycos
x y cos
T
T
y yy y
x xx x
yx yx
x y
x y
x y
• Angle between two vectors
Orthogonal vectors: xT y = 0
x
y
=/2
||x||||y||
y2
y1
x
Some Vector Concepts
![Page 6: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/6.jpg)
6
Linear Map and Matrices
Linear mappings are almost omnipresent
If both domain and co-domain are both finite-dimensional vector space, each linear mapping can be uniquely represented by a matrix w.r.t. specific couple of bases
We intend to study properties of linear mapping from properties of its matrix
![Page 7: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/7.jpg)
7
Linear map and MatricesT
his iso
mo
rph
ism is b
asis d
epen
den
t
![Page 8: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/8.jpg)
8
Linear map and Matrices
Let A be similar to B, i.e. B=P-1AP
Similarity defines an equivalent relation in the vector space of square matrices of orde n, i.e. it partitions the vector space in to different equivalent classes.
Each equivalent class represents unique linear operator
How can we choose
i) the simplest one in each equivalent class and
ii) The one of special interest ??
![Page 9: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/9.jpg)
9
A major concern of ours is to make the best choice of basis, so that the linear operator with which we are working will have a representing matrix in the chosen basis that is as simple as possible.
Linear map and Matrices
Two matrices representing the same linear transformation with respect to different bases must be similar.
A diagonal matrix is a very useful matrix, for example,
Dn=P-1AnP
![Page 10: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/10.jpg)
10
Linear map and Matrices
Each equivalent class represent unique linear operator
Can we characterize the class in simpler way?
Yes, we can Under extra conditions
The concept , characteristic roots plays an important role in this regards
![Page 11: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/11.jpg)
11
Definition: The quadratic form in n variables x1, x2, …, xn is the general homogeneous function of second degree in the variables
n
i
n
jjiijn xxaxxxfY ),...,,( 21
Axx
x
x
x
aaa
aaa
aaa
xxxY T
nnnnn
n
n
n
...
...
............
...
...
... 2
1
21
22221
11211
21
In terms of matrix notation, the quadratic form is given by
Quadratic Form
![Page 12: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/12.jpg)
12
Examples of Some Quadratic Forms
1
2.
3.
2221
21 762 xxxxAxxY T
13322123
22
21 1283021121 xxxxxxxxxAxxY T
2222
211 .... nn
T xaxaxaAxxY
A can be many for a particular quadratic form. To make it unique it is customary to write A as symmetric matrix.
Standard form What is its uses??
![Page 13: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/13.jpg)
13
In Fact Infinite A’s
• For example 1 we have to take a12, and a21 s.t.a12 +a21 =6.
• We can do it in infinite ways.
• Symmetric A
![Page 14: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/14.jpg)
14
Its Importance in Statistics
Variance is a fundamental concept in statistics. It is nothing but a quadratic form with a idempotent matrix of rank (n-1)
Quadratic forms play a central role in multivariate statistical analysis. For example, principal component analysis, factor analysis, discriminant analysis etc.
Tn
Tn
ii
nIAwhere
Axxn
xxn
111
,
;...1
)(1
1
2
![Page 15: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/15.jpg)
15
Multivariate Gaussian
0
Its Importance in Statistics
![Page 16: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/16.jpg)
16
Bivariate Gaussian
Its Importance in Statistics
![Page 17: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/17.jpg)
17
Spherical, diagonal, full covariance
UM
![Page 18: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/18.jpg)
18
Quadratic Form as Inner Product
XT AX=(AT X)TX = XT (AX) = XT Y
• Let A=CTC.
Then XT AX= XTCTCX=(CX)TCX=YTY
XT AY= XTCTCY=(CX)TCY=WT Z
What is its geometric meaning ?
Different nonsingular Cs represent different inner products
Different inner products different geometries.
Length ofY, ||Y||= (YTY)1/2;
XTY, dot product of X andY
![Page 19: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/19.jpg)
19
Euclidean Distance and Mathematical Euclidean Distance and Mathematical DistanceDistance
• Usual human concept of distance is Eucl. Dist.Usual human concept of distance is Eucl. Dist.• Each coordinate contributes equally to the distanceEach coordinate contributes equally to the distance
2222
211
2121
)()()(),(
),,,(),,,,(
pp
pp
yxyxyxQPd
yyyQxxxP
Mathematicians, generalizing its three properties ,
1) d(P,Q)=d(Q,P).2) d(P,Q)=0 if and only if P=Q and
3) d(P,Q)=<d(P,R)+d(R,Q) for all R, define distance on any set.
![Page 20: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/20.jpg)
20
Statistical DistanceStatistical Distance• Weight coordinates subject to a great deal of variability Weight coordinates subject to a great deal of variability
less heavily than those that are not highly variableless heavily than those that are not highly variable
Who is nearer to data set if it
were point?
![Page 21: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/21.jpg)
21
Statistical Distance for Uncorrelated DataStatistical Distance for Uncorrelated Data
2 2
2 2* * 1 21 2
11 22
( , )x x
d O P x xs s
1 2( , ), (0,0)P x x O
* *1 1 11 2 2 22
/ , /x x s x x s
![Page 22: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/22.jpg)
22
Ellipse of Constant Statistical Distance for Ellipse of Constant Statistical Distance for Uncorrelated DataUncorrelated Data
11sc 11sc
22sc
x1
x2
0
22sc
![Page 23: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/23.jpg)
23
Scattered Plot for Scattered Plot for Correlated MeasurementsCorrelated Measurements
![Page 24: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/24.jpg)
24
Statistical Distance under Rotated Statistical Distance under Rotated Coordinate SystemCoordinate System
2 211 1 12 1 2 22 2
( , ) 2d O P a x a x x a x
1 2
2 21 2
11 22
(0,0), ( , )
( , )
O P x x
x xd O P
s s
1 1 2
2 1 2
cos sin
sin cos
x x x
x x x
![Page 25: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/25.jpg)
25
General Statistical DistanceGeneral Statistical Distance
)])((2
))((2))((2
)(
)()([
),(
]222
[),(
),,,(),0,,0,0(),,,,(
11,1
331113221112
2
22222
21111
1,131132112
22222
2111
2121
pppppp
pppp
pppp
ppp
pp
yxyxa
yxyxayxyxa
yxa
yxayxa
QPd
xxaxxaxxa
xaxaxaPOd
yyyQOxxxP
![Page 26: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/26.jpg)
26
Necessity of Statistical DistanceNecessity of Statistical Distance
![Page 27: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/27.jpg)
27
Mahalonobis Distance
)()(),,( 1 μxΣμxΣμx TMDPopulation version:
Sample veersion;
We can robustify it using robust estimators of location and scatter functional
)()(),,( 1 xxSxxSxx TMD
![Page 28: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/28.jpg)
28
Classification of Quadratic Form
• Chart: Quadratic Form
Definite Indefinite
Positive Definite
Positive Semi definite
Negative Definite
Negative Semi definite
![Page 29: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/29.jpg)
29
Classification of Quadratic FormDefinitions
1. Positive Definite: A quadratic form Y=XTAX is said to be positive definite iff Y=XTAX>0 ; for all x≠0 . Then the matrix A is said to be a positive definite matrix.
2. Positive Semi-definite:A quadratic form, Y=XTAX is said to be positive semi-definite iff Y=XTAX>=0 , for all x≠0 and there exists x≠0 such that XTAX=0 . Then the matrix A is said to be a positive semidefinite matrix.
3. Negative Definite: A quadratic form Y=XTAX is said to be negative definite iff Y=XTAX<=0 for all x≠0. Then the matrix A is said to be negative definite matrix
![Page 30: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/30.jpg)
30
4. Negative Semi-definite: A quadratic form,
is said to be negative semi-definite iff , for all x≠0 and there exists x≠0 such that . The matrix A is said to be a negative semi-definite matrix.
Indefinite: Quadratic forms and their associated symmetric matrices need not be definite or semi-definite in any of the above scenes. In this case the quadratic form is said to be indefinite; that is , it can be negative, zero or positive depending on the values of x.
AxxY T0 AxxY T
0AxxT
Classification of Quadratic FormDefinitions
![Page 31: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/31.jpg)
31
Two Theorems On Quadratic Form
Theorem(1): A quadratic form can always be expressed with respect to a given coordinate system as . where A is a unique symmetric matrix.
Theorem2: Two symmetric matrices A and B represent the same quadratic form if and only if
B=PTAP where P is a non-singular matrix.
AxxY T
![Page 32: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/32.jpg)
32
Classification of Quadratic FormImportance of Standard Form
From standard form we can easily classify a quadratic form.
XT AX=
Is positive /positive semi/negative/ negative semidifinite/indefinite if ai >0 for all i/ ai >0 for some i others, a=0/ai <0 for all i,/ ai <0 some i , others, a=0/ Some ai are +ive, some are negative.
2
1i
n
iixa
![Page 33: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/33.jpg)
33
That is why using suitable nonsingular trandformation ( why nonsingular??) we try to transform general XT AX into a standard form.If we can find a P nonsingular matrix s.t.
we can easily classify it. We can do it i) for congruent transformation and ii) using eigenvalues and eigen vectors.
matrix diagonal aD,
DAPPT
Classification of Quadratic FormImportance of Standard Form
Method 2 is mostly used in MM
![Page 34: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/34.jpg)
34
1. Positive Definite: (a). A quadratic form is positive definite iff the nested principal minors of A is given as
Evidently a matrix A is positive definite only if det(A)>0
(b). A quadratic form Y=XTAX be positive definite iff all the eigen values of A are positive.
0 AxxY T
0
...
............
...
,........,0,0,0
21
2....2221
11211
333231
232221
131211
2221
121111
nnnn
n
n
aaa
aaa
aaa
aaa
aaa
aaa
aa
aaa
Classification of Quadratic FormImportance of Determinant, Eigen Values and
Diagonal Element
![Page 35: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/35.jpg)
35
2. Positive Semi-definite:(a) A quadratic form is positive semi-definite iff the nested principal minors of A is given as
(b). A quadratic form Y=XTAX is positive semi-definite iff at least one eigen value of A is zero while the remaining roots are positive.
AxxY T
Classification of Quadratic FormImportance of Determinant, Eigen Values and
Diagonal Element
0
...
............
...
,........,0,0
21
2....2221
11211
2221
121111
nnnn
n
n
aaa
aaa
aaa
aa
aaa
![Page 36: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/36.jpg)
36
Continued
3. Negative Definite: (a). A quadratic form is negative definite iff the nested principal minors of A are given as
Evidently a matrix A is negative definite only if (-1)n× det(A)>0; where det(A) is either negative or positive depending on the order n of A.
(b). A quadratic form Y=XTAX be negative definite iff all the eigen Roots of A are negative.
AxxY T
,........0,0,0
333231
232221
131211
2221
121111
aaa
aaa
aaa
aa
aaa
![Page 37: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/37.jpg)
37
4. Negative Semi-definite:(a)A quadratic form is negative semi-definite iff the nested principal minors of A is given as
Evidently a matrix A is negative semi-definite only if
,that is, det(A)≥0 ( det(A)≤0 ) when n is odd( even).
(b). A quadratic form is negative semi-definite iff at least one eigen value of A is zero while the remaining roots are negative.
AxxY T
AxxY T
0)1( An
,........0,0,0
333231
232221
131211
2221
121111
aaa
aaa
aaa
aa
aaa
Continued
0)1( An
![Page 38: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/38.jpg)
38
Theorem on Quadratic Form(Congruent Transformation)
If is a real quadratic form of n variables x1, x2, …, xn and rank r i.e. ρ(A)=r then there exists a non-singular matrix P of order n such that x=Pz will convert Y in the canonical form
where λ1, λ2, …, λr are all the different from zero.
That implies
AxxY T
2222
211 ... rr zzz
}00,,,diag{D , 1 rT
TTT
DZZ
APZPZAXX
![Page 39: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/39.jpg)
39
Grammian (Gram)Matrix
Grammian Matrix
-----If A be n×m matrix then the matrix S=ATA is called grammian matrix of A. If A is m×n then S=ATA is a symmetric n-rowed matrix.
Properties
a. Every positive definite or positive semi-definite matrix can be represented as a Grammian matrix
b. The Grammian matrix ATA is always positive definite or positive semi-definite according as the rank of A is equal to or less than the number of columns of A
c.
d. If ATA=0 then A=0
rAAAArA TT )()()(
![Page 40: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/40.jpg)
40
What are eigenvalues?
• Given a matrix, A, x is the eigenvector and is the corresponding eigenvalue if Ax = x– A must be square and the determinant of A - I must
be equal to zeroAx - x = 0 ! (A - I) x = 0
• Trivial solution is if x = 0• The non trivial solution occurs when det(A - I) = 0
• Are eigenvectors are unique?– If x is an eigenvector, then x is also an eigenvector
and is an eigenvalue of A,A(x) = (Ax) = (x) = (x)
![Page 41: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/41.jpg)
41
Calculating the Eigenvectors/values
• Expand the det(A - I) = 0 for a 2 × 2 matrix
• For a 2 × 2 matrix, this is a simple quadratic equation with two solutions (maybe complex)
• This “characteristic equation” can be used to solve for x
0
00det
010
01detdet
2112221122112
211222112221
1211
2221
1211
aaaaaa
aaaaaa
aa
aa
aaIA
}4{2
121122211
222112211 aaaaaaaa
![Page 42: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/42.jpg)
42
Eigenvalue example• Consider,
• The corresponding eigenvectors can be computed as
– For = 0, one possible solution is x = (2, -1)– For = 5, one possible solution is x = (1, 2)
5,0)41(
02241)41(
0
42
21
2
2211222112211
2
aaaaaa
A
0
0
12
24
12
240
50
05
42
215
0
0
42
21
42
210
00
00
42
210
yx
yx
y
x
y
x
yx
yx
y
x
y
x
![Page 43: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/43.jpg)
43
Geometric Interpretation of eigen roots and vectors
We know from the definition of eigen roots and vectors Ax = λx; (**)
where A is m×m matrix, x is m tuples vector and λ is scalar quantity.
From the right side of (**) we see that the vector is multiplied by a scalar. Hence the direction of x and λx is on the same line.
The left side of(**)shows the effect of matrix multiplication of matrix A (matrix operator) with vector x. But matrix operator may change the direction and magnitude of the vector.
![Page 44: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/44.jpg)
44
Geometric Interpretation of eigen roots and vectors
Hence our goal is to find such kind of vectors that change in magnitude but remain on the same line after
matrix multiplication. Now the question arises: does these eigen vectors along
with their respective change in magnitude characterize the matrix?
Answer is the DECOMPOSITION THEOREMS
![Page 45: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/45.jpg)
45
Geometric Interpretation of eigen Roots and Vectors
Y
X
x1
x2
[A] Y
X
Ax1
Geometric Interpretation
Ax2
ZZ
![Page 46: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/46.jpg)
46
More to Notice
1 1
2 2
0
0
a x xa
a x x
1 1
2 2
1 1
2 2
0 0 0 0,
0 b 0 0 0 b
0
0 b
a x x aa b
x x
a x ax
x bx
![Page 47: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/47.jpg)
47
Properties of Eigen Values and Vectors
If B=CAC-1, where A, B and C are all n×n then A and B have same eigen roots. If x is the eigen Vector of A then Cx for BThe eigen roots of A and AT are same.A eigen Vector x≠o can not be associated with more than one eigen Root The eigen Vectors of a matrix A are linearly independent if they corresponds to distinct roots.Let A be a square matrix of order m and suppose all its roots are distinct. Then A is similar to a diagonal matrix Λ,i.e. P-1AP= Λ.eigen Roots and vectors are all real for any real symmetric matrix, A
If λi and λj are two distinct roots of a real symmetric matrix A, then vectors xi and xj are orthogonal
![Page 48: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/48.jpg)
48
If λ1, λ2, … , λm are the eigen roots of the non-singular matrix A then λ1
-1, λ2-1, … , λm
-1 are the eigen roots of A-1.
Let A, B be two square matrices of order m. Then the eigen roots of AB are exactly the eigen roots of BA.
Let A, B be respectively m×n and n×m matrices, where m≤n. Then the eigen Roots of (BA)n×n consists of n-m zeros and the m eigen Roots of (AB)m×m.
Properties of Eigen Values and Vectors
![Page 49: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/49.jpg)
49
Let A be a square matrix of order m and λ1, λ2, … , λm be its eigen Roots then .
Let A be a m×m matrix with eigen Roots λ1, λ2, … , λm then tr(A) = tr(Λ) = λ1+ λ2+ … + λm .
If A has eigen Roots λ1, λ2, … , λm then A-kI has eigen Roots λ1-k, λ2-k, … , λm-k and kA has the eigen Roots kλ1, kλ2, … , kλm , where k is scalar.
If A is an orthogonal matrix then all its eigen Roots have absolute value 1.
Let A be a square matrix of order m; suppose further that A is idempotent. Then its eigen Roots are either 0 or 1.
m
ii
A1
Properties of Eigen Values and Vectors
![Page 50: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/50.jpg)
50
![Page 51: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/51.jpg)
51
Eigen/diagonal Decomposition
• Let be a square matrix with m linearly independent eigenvectors (a “non-defective” matrix)
• Theorem: Exists an eigen decomposition
– (cf. matrix diagonalization theorem)
• Columns of U are eigenvectors of S
• Diagonal elements of are eigenvalues of
diagonalUnique for
distinct eigen-values
![Page 52: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/52.jpg)
52
Diagonal decomposition: why/how
mvvU ...1Let U have the eigenvectors as columns:
m
mmmm vvvvvvSSU
............
1
1111
Then, SU can be written
And S=UU–1.
Thus SU=U, or U–1SU=
UM
![Page 53: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/53.jpg)
53
Diagonal decomposition - example
Recall .3,1;21
1221
S
The eigenvectors and form
1
1
1
1
11
11U
Inverting, we have
2/12/1
2/12/11U
Then, S=UU–1 =
2/12/1
2/12/1
30
01
11
11
RecallUU–1 =1.
UM
![Page 54: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/54.jpg)
54
Example continued
Let’s divide U (and multiply U–1) by 2
2/12/1
2/12/1
30
01
2/12/1
2/12/1Then, S=
Q (Q-1= QT )
Why? …
![Page 55: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/55.jpg)
55
Symmetric Eigen Decomposition
• If is a symmetric matrix:
• Theorem: Exists a (unique) eigen decomposition
• where Q is orthogonal:
– Q-1= QT
– Columns of Q are normalized eigenvectors
– Columns are orthogonal.
– (everything is real)
TQQS
![Page 56: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/56.jpg)
56
Spectral Decomposition theorem• If A is a symmetric m ×m matrix with i and ei,
i = 1 m being the m eigenvector and eigenvalue pairs, then
– This is also called the eigen( spectral) decomposition theorem
• Any symmetric matrix can be reconstructed using its eigenvalues and eigenvectors
Tm
i m
Ti
mii
mmm
Tm
mmm
m
T
mm
T
mmmPPeeAeeeeeeA
1 111112
122
11
111
m
mmm
mm
00
00
00
,, 2
1
21 eeeP
![Page 57: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/57.jpg)
57
Example for Spectral Decomposition
• Let A be a symmetric, positive definite matrix
• The eigenvectors for the corresponding eigenvalues are
• Consequently,
02316.016.65
0det8.24.0
4.02.2
2
IAA
51,
52,
52,
51
21TT ee
4.08.0
8.06.1
4.22.1
2.16.0
51
52
51
52
25
25
1
52
51
38.24.0
4.02.2A
![Page 58: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/58.jpg)
58
The Square Root of a MatrixThe spectral decomposition allows us to express a
square matrix in terms of its eigenvalues and eigenvectors.
This expression enables us to conveniently create a square root matrix.A is a p x p positive definite matrix with the spectral
decomposition:
k
'i i i
i 1
A ee P = [ e1 e2 e3 … ep]
k
' 'i i i
i 1
A ee P P
where P’P = PP’ = I and = diag(i).
If
![Page 59: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/59.jpg)
59
The Square Root of a Matrix
112 2
p
0 00 0
0 0
This implies (P1/2P’)PP’ = P1/2P’ = (PP’)
Let
The matrix
1 1k
' '2 2i i i
i 1
P P ee AIs called he square root of A
![Page 60: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/60.jpg)
60
Matrix Square Root Properties
The square root of A has the following properties(prove them):
'1 12 2A A
1 12 2A A A
-1 1 1 -12 2 2 2A A A A I
-1 -1 -1 112 2 2 2 where
-1
A A A A A
![Page 61: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/61.jpg)
61
Physical Interpretation of SPD(Spectral Decomposition)
e1e2
Suppose xT Ax = c2. For p = 2, all x that satisfy this equation form an ellipse, i.e., c2 = 1(xTe1)2
+ 2(xT e2)2 (using SPD of p.d. A).
Let x1 = c 1-1/2 e1 and x2 = c λ2
-1/2 e2. Both x satisfy the above equation in the direction of eigenvector. Note that the length of x is c 1
-1/2. ||x|| is inversely propotional to sqrt of eigen values of A.
Wh
at will b
e the case
if we rep
lace A b
y A-1
?
Var(eT
i
x)=e
iT
Var(X
)eT=
Λi ith
eigen
value
ofV
ar(X)
All points at same ellipse-distance.
![Page 62: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/62.jpg)
62
Matrix Inequalitiesand Maximization
- Extended Cauchy-Schwartz Inequality – Let b and d be any two p x 1 vectors and B be a p x p positive definite matrix. Then
(b’d)2 (b’Bb)(d’B-1d)
with equality iff b=kB-1d or (or d=kB -1d) for some constant c.
- Maximization Lemma – let d be a given p x 1 vector and B be a p x p positive definite matrix. Then for an arbitrary nonzero vector x
2
1'
'max'x 0
x dd B d
x Bx
with the maximum attained when x = kB-1d for any constant
k.
![Page 63: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/63.jpg)
63
Matrix Inequalitiesand Maximization
- Maximization of Quadratic Forms for Points on the Unit Sphere – let B be a p x p positive definite matrix with eigenvalues 1 2 p and associated eigenvectors e1, e2, ,ep. Then
1 1
p p
' (attained when = )max
''
(attained when = )min'
x 0
x 0
x Bxx e
x xx Bx
x ex x
1 k
k+1 k+1,
' (attained when , k 1,2, ,p-1)max
'x e e
x Bxx e
x x
![Page 64: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/64.jpg)
64
t<-sqrt(2)x<-c(3.0046,t,t,16.9967)A<-matrix(x, nrow=2)eigen(A)
> x[1] 3.004600 1.414214 1.414214 16.996700> A<-matrix(x, nrow=2)> A [,1] [,2][1,] 3.004600 1.414214[2,] 1.414214 16.996700> eigen(A)$values[1] 17.138207 2.863093
$vectors [,1] [,2][1,] 0.09956317 -0.99503124[2,] 0.99503124 0.09956317
Calculation in R
![Page 65: 1 Quadratic Forms, Characteristic Roots and Characteristic Vectors Mohammed Nasser Professor, Dept. of Statistics, RU,Bangladesh Email: mnasser.ru@gmail.com](https://reader036.vdocuments.us/reader036/viewer/2022062716/56649de75503460f94ae1746/html5/thumbnails/65.jpg)
65
Thank you