statistics and linear algebra (the real thing). vector a vector is a rectangular arrangement of...
TRANSCRIPT
Statistics and Linear AlgebraStatistics and Linear Algebra
(the real thing)(the real thing)
VectorVector
A vector is a rectangular arrangement of number in several rows and one column. A vector is denoted by a lowercase bold font weight: x
A vector element is given by the value (scalar) of a given row. x1=6 and x2=1
The number of elements gives its dimensionality Since there are two elements, the dimensionality of x is 2. (Each elements can be considered as a subject score)
The transpose of vector, denoted T (or `), is a rotation of the column and rows.
6
1
x
Definitions
TT 6 1x
VectorVector If the dimensionality of a vector is equal or less than 3, it can be represented graphically A vector is represented as an arrow (orientation and length)
6
1
x
VectorVector
1.5
6 91.5
1 1.5
s
s
x
Operations: scalar multiplication When a vector is multiply by a scalar, each element of the vector is multiply by the scalar. When a vector is multiply by a scalar (number), then its length is increased by the factor of the scalar.
VectorVector
1
6 61
1 1
s
s
x
When a vector is multiply by -1, then it will reverse is direction.
Operations: -1 multiplication
VectorVector
6 2 and
1 5
6 2 8 +
1 5 6
x y
x y
When two vectors are added together, we sum their corresponding elements. Graphically, we put the beginning of the second vector at the end of the first.
Operations: Addition of two vectors
VectorVector
6 2 and
1 5
6 2 6 2 4 - +
1 5 1 5 4
x y
x y
When two vectors are added together, we sum their corresponding elements. Graphically, we put the beginning of the second vector at the end of the first one, once the second vector has been multiply by -1.
Operations: Subtraction of two vectors
What 2x-3y will gives ?
MatricesMatrices
A matrix can be view as a collection of vectors. A matrix is denoted by an uppercase bold font weight: M The matrix dimension is given by its number of rows and columns
Example: 2 rows and 4 columns: 24 A vector can be view as a matrix with many rows and one column. A matrix is called “square”, if n×n. A matrix element is given by the junction of the given row and column, denoted at Mij
Definitions
1 2 5 6
2 4 5 6
M 13 5M
MatricesMatrices
When a matrix is multiply by a scalar, each element is multiply by the scalar.
Operation: multiplication of matrix by a scalar
1.5
1 2 5 6 1.5 3 7.5 91.5
2 4 5 6 3 6 7.5 9
s
s
M
If two vectors have the same dimension, then they can be multiply together There will be two possible results: a) A scalar or b) A matrix
Scalar (inner product, dot product) Two vectors will output a scalar, if the first vector is transposed before being multiplied with the second vector (of equal dimension). The row of the first vector is multiplied by the corresponding element of the second vector, and the resulting products are sum up.
MatricesMatricesOperation: Product of two vectors
1 1
2 2
3 3
1TT
1 2 3 2 1 1 2 2 3 3
3
and ;
x y
x y
x y
y
x x x y x y x y x y
y
x y
x y
T
4 3 and
6 5
?
x y
x y
If we divided xTy by their corresponding degrees of freedom (n-1) we obtain the covariance between the two variables (if the mean is zero).
MatricesMatrices
Matrix (outer product) Two vectors will output a matrix, if the second vector is transposed before being multiplied. The column of first vector is multiplied by the corresponding element of the second
vector row.
Operation: Product of two vectors
1 1
2 2
3 3
1 1 1 1 2 1 3TT
2 1 2 3 2 1 2 2 2 3
3 3 1 3 2 3 3
and ;
x y
x y
x y
x x y x y x y
x y y y x y x y x y
x x y x y x y
x y
xy
T
4 3 and
6 5
?
x y
xy
MatricesMatrices
Two matrices can be multiplied together, if the number of columns of the first matrix is equal to the number of rows of the second matrix. Ex: If A is a m3 matrix, then B must be a 3n matrix. The resulting matrix C will be a mn matrix
The matrix product is not commutative: ABBA
Operation: Product of two matrices
3 12 3 1
and 4 2 ;1 4 0
5 3
3 12 3 1 (2 3) ( 3 4) (1 5) (2 1) ( 3 2) (1 3) 1 7
4 21 4 0 ( 1 3) (4 4) (0 5) ( 1 1) (4 2) (0 3) 13 7
5 3
A B
C AB
9 8 7
1 2 3 4 6 5 4 and ; ?
5 6 7 8 3 2 1
0 9 8
X Y XY
MatricesMatrices
There is a special kind of matrix that is similar to the arithmetic multiplication by one. 51=5 This matrix is called: Identity, denoted by I, where all its diagonal elements are set to one and the remaining elements to 0. Since this matrix has the same number of columns and rows: AI=A or IA=A
Identity matrix
1 0 0 0 1 2 3 4
0 1 0 0 5 6 7 8 and = ;
0 0 1 0 9 0 1 2
0 0 0 1 3 4 5 6
1 0 0 0 1 2 3 4 1 2 3 4
0 1 0 0 5 6 7 8 5 6 7 8
0 0 1 0 9 0 1 2 9 0 1 2
0 0 0 1 3 4 5 6 3 4 5 6
A
IA
MatricesMatrices
A vector whose all elements are equal to 1. It is denoted by 1
Addition-Operator Vector
30 15
25 101
and 28 121
32 14
22 13
30 15 45
25 10 351
28 12 401
32 14 46
22 13 35
X 1
X1
1
1
1
1
1
MatricesMatrices
If the norm is divided by the degrees of freedom (n-1), then the standard deviation (if the mean is zero) is obtained.
The Norm-Operation of a vector
2 2By Pythagoras, 6 1 37 6.08276 x
6
1
T 6In vector notation, 6 1 37
1
x x x
6
1
x
MatricesMatrices
Is a function that associates a scalar, det(A), to every n×n square matrix A. This can be interpreted as the volume of the matrix. In 2D, the area of the parallelogram
The Determinant of Matrix
4 3
2 5
A
MatricesMatrices
Is a function depending on n that associates a scalar, det(A), to every n×n square matrix A. S4 = the area of the rectangle
The Determinant of Matrix
4 3
2 5
A
T1
T1
S3
T2
S3S1 T1 T1
S2 T2 T2
Area S4 2T1 2T2 2S3
Area S4 S1 S2 2S3
Area (4 2)(5 3) 4 3
2 5 2 (2 3) 14
T2
MatricesMatrices
Is a function depending on n that associates a scalar, det(A), to every n×n square matrix A. S4 = the area of the rectangle
The Determinant of Matrix
a b
c d
A
T1
T1
S3
T2
S3S1 T1 T1
S2 T2 T2
Area S4 2T1 2T2 2S3
Area S4 S1 S2 2S3
Area (a+c)(b+d)-ab-cd-2bc
Area ab+ad+bc+cd-ab-cd-2bc
Area ad-bc
det( ) 4 5 2 3 14 A
4 3
2 5
AT2
MatricesMatrices
In analogy with the exponential notation for reciprocal of a number (1/a=a -1), the inverse of matrix is denoted A-1. If a matrix is square, then A-1A=AA-1=I. For a 22 matrix, the operation is
The Inverse of Matrix
1
;
1 1
det( )
a b
c d
d b d b
c a c aad bc
A
AA
1
1
1 4;
3 2
2 41 1
3 12 12 10
0.2 0.4
0.3 0.1
d b
c a
A
A
A1
1
1 4 0.2 0.4 1 0
3 2 0.3 0.1 0 1
0.2 0.4 1 4 1 0
0.3 0.1 3 2 0 1
AA
A A
Linear Algebra and StatisticsLinear Algebra and Statistics
A vector is normalized if its length is equal to one. Normalizing a vector = data standardization.
The Normalization of Vector
3; 25 5
4
3
0.64
0.85
x x
z
T 1
xz
x
z z
Linear Algebra and StatisticsLinear Algebra and Statistics
If two variables (u and v) have the same score, then the two vectors are superposed on each other. However, as the two variables differs from one another, the angle between them will increase.
Relation between two vectors
Linear Algebra and StatisticsLinear Algebra and Statistics
The greater the angle between the two vectors, the lesser they share in common. If the angle reach 90° then there are no common part.
Relation between two vectors
Linear Algebra and StatisticsLinear Algebra and Statistics
The cosine of that angle is the correlation coefficient. If the angle is null (or 180°) then the cosine is 1 (or -1); indicating a perfect
relation. If the relation is 90° (or 270 °) then the cosine is 0; indication an absence of relation.
Relation between two vectors
T1 cov
cos
n
i ii r
s s
uv
uvu v
u vu v
u v u v
Linear Algebra and StatisticsLinear Algebra and StatisticsRelation between two vectors
T cosu v u v
6 2 and
1 5
u v
T 17u v
37 and 29 u v
T 17cos 0.52
37 29r uv
u v
u v
T1 cov
cos
n
i ii r
s s
uv
uvu v
u vu v
u v u v
Linear Algebra and StatisticsLinear Algebra and StatisticsThe Mean
T / nx 1 X30
25
, ?28
32
22
X x
1
1
30 25 28 32 22 / 51
1
1
27.4
x
x
From the previously defined addition-operation, the mean is straightforward. Let us say that we have one variable with 5 participants.
Linear Algebra and StatisticsLinear Algebra and StatisticsThe Mean (several variables)
30 15
25 10
, [27.4 12.8]28 12
32 14
22 13
M x
Let us say that we have two variables with 5 participants.
Let us define a 52 mean-score matrix as
27.4 12.8
27.4 12.8
27.4 12.8
27.4 12.8
27.4 12.8
X
Linear Algebra and StatisticsLinear Algebra and Statistics
Example of statistical Import
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
X M X
Deviation Score Matrix
30 15 27.4 12.8
25 10 27.4 12.8
, 28 12 27.4 12.8
32 14 27.4 12.8
22 13 27.4 12.8
30 15 27.4 12.8 2
25 10 27.4 12.8
28 12 27.4 12.8
32 14 27.4 12.8
22 13 27.4 12.8
M X
X M X
.6 2.2
2.4 2.8
0.6 0.8
4.6 1.2
5.4 0.2
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
T , where is a deviation score matrixSSCP X X X
Sums of Square and Cross Product (SSCP)
T
2.6 2.2
2.4 2.8
0.6 0.8
4.6 1.2
5.4 0.2
2.6 2.2
2.4 2.82.6 2.4 0.6 4.6 5.4 63.2 16.4
0.6 0.82.2 2.8 0.8 1.2 0.2 16.4 14.8
4.6 1.2
5.4 0.2
X
SSCP X X
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
Sums of Square and Cross Product (SSCP) Note: the SSCP is square and symmetric If the SSCP is divided by the number of degrees of freedom, then we get the information about variance and covariance for all the data.
63.2 16.4
16.4 14.8
SSCP
63.2 16.4 15.8 4.1/(5 1)
16.4 14.8 4.1 3.7
VARCOV
1s
2s
1s 2s
Variance for the first variable
Variance for the second variable
Covariance
21 1 1( )s s s
22 2 2( )s s s
1 2 2 1 12( )s s s s Cov
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
Simple regression How can we find the weights that describe (optimally) the following function ?
0 1v̂ b b u
The solution is to find the shadow of v on u that has the shortest distance
The shortest distance is the one that crosses at 90° the vector u
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
Simple regression Therefore, the error e can be defined as:
1b e v u
Where b1 is the value that multiply u that makes the shadow of v the shortest (90°)
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
Simple regression As a consequence, the angle between u and e will also be 90° Therefore, the correlation (and covariance) between u and e will be zero.
T 0u e
By substitution, we can isolate the b1 coefficient.
T
T1
T T1
T T1
T 1 T T 1 T1
T 1 T1 1
0
( ) 0
0
( ) ( ) ( ) ( )
( ) ( ) 1
b
b
b
b
b b
u e
u v u
u v u u
u v u u
u u u v u u u u
u u u v
This is the least mean squared method
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
Simple regression With 2 variables this is identical to the solution given in textbooks.
-1 -1T T 21 2
covcovb s
s uv
u uvu
u u u v 2.6 2.2
2.4 2.8
0.6 0.8
4.6 1.2
5.4 0.2
X
u v
Deviation score matrix
-1
11
2.6 2.2
2.4 2.8
2.6 2.4 0.6 4.6 5.4 2.6 2.4 0.6 4.6 5.4 63.2 16.4 0.260.6 0.8
4.6 1.2
5.4 0.2
b
Linear Algebra and StatisticsLinear Algebra and StatisticsExample of statistical Import
Simple regression The constant b0 is obtained the usual way
0 1
0 1
v b b u
b v b u
0
27.4 and 12.8
12.8 0.26 27.4 5.69
u v
b
Therefore, the final regression equation is
ˆ 5.69 0.26v u