Download - 23 Matrix Algorithms
![Page 1: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/1.jpg)
Analysis of AlgorithmsMatrix algorithms
Andres Mendez-Vazquez
November 22, 2015
1 / 102
![Page 2: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/2.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
2 / 102
![Page 3: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/3.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
3 / 102
![Page 4: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/4.jpg)
Basic definitions
A matrix is a rectangular array of numbers
A =(a11 a12 a13a21 a22 a23
)=(
1 2 34 5 6
)
A transpose matrix is the matrix obtained by exchanging the rows andcolumns
AT =
1 42 53 6
4 / 102
![Page 5: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/5.jpg)
Basic definitions
A matrix is a rectangular array of numbers
A =(a11 a12 a13a21 a22 a23
)=(
1 2 34 5 6
)
A transpose matrix is the matrix obtained by exchanging the rows andcolumns
AT =
1 42 53 6
4 / 102
![Page 6: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/6.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
5 / 102
![Page 7: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/7.jpg)
Several cases of matrices
Zero matrix 0 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 0
The diagonal matrix
a11 0 · · · 00 a22 · · · 0...
... . . . ...0 0 · · · ann
6 / 102
![Page 8: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/8.jpg)
Several cases of matrices
Zero matrix 0 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 00 0 0 0 0
The diagonal matrix
a11 0 · · · 00 a22 · · · 0...
... . . . ...0 0 · · · ann
6 / 102
![Page 9: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/9.jpg)
Several cases of matrices
Upper triangular matrixa11 a12 · · · a1n0 a22 · · · a2n...
... . . . ...0 0 · · · ann
7 / 102
![Page 10: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/10.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
8 / 102
![Page 11: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/11.jpg)
Operations on matrices
They Define a Vectorial SpaceMatrix addition.Multiplication by scalar.The existence of zero.
9 / 102
![Page 12: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/12.jpg)
Operations on matrices
They Define a Vectorial SpaceMatrix addition.Multiplication by scalar.The existence of zero.
9 / 102
![Page 13: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/13.jpg)
Operations on matrices
They Define a Vectorial SpaceMatrix addition.Multiplication by scalar.The existence of zero.
9 / 102
![Page 14: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/14.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
10 / 102
![Page 15: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/15.jpg)
Matrix Multiplication
What is Matrix Multiplication?Given A, B matrices with dimensions n×n, the multiplication is defined as
C = AB
cij =n∑k=1
aikbkj
11 / 102
![Page 16: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/16.jpg)
Complexity and Algorithm
Algorithm: Complexity Θ (n3)Square-Matrix-Multiply(A,B)
1 n = A.rows
2 let C be a new matrix n× n3 for i = 1 to n4 for j = 1 to n5 C [i, j] = 06 for k = 1 to n7 C [i, j] = C [i, j] +A [i, j] ∗B [i, j]8 return C
12 / 102
![Page 17: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/17.jpg)
Matrix multiplication properties
Properties of the MultiplicationThe Identity exist for a matrix A of m× n:
ImA = AIn = A.
The multiplication is associative:
A(BC) = (AB)C.
In addition, multiplication is distibutiveA(B + C) = AB +AC
(B + C)D = BD + CD
13 / 102
![Page 18: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/18.jpg)
Matrix multiplication properties
Properties of the MultiplicationThe Identity exist for a matrix A of m× n:
ImA = AIn = A.
The multiplication is associative:
A(BC) = (AB)C.
In addition, multiplication is distibutiveA(B + C) = AB +AC
(B + C)D = BD + CD
13 / 102
![Page 19: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/19.jpg)
Matrix multiplication properties
Properties of the MultiplicationThe Identity exist for a matrix A of m× n:
ImA = AIn = A.
The multiplication is associative:
A(BC) = (AB)C.
In addition, multiplication is distibutiveA(B + C) = AB +AC
(B + C)D = BD + CD
13 / 102
![Page 20: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/20.jpg)
In addition
DefinitionThe inner product between vectors is defied as
xT y =n∑i=1
xiyi
14 / 102
![Page 21: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/21.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
15 / 102
![Page 22: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/22.jpg)
Matrix inverses
The inverse is defined as the vector A−1 such that
AA−1 = A−1A = In
Example(1 11 0
)−1=(
0 11 −1
)=⇒
(1 11 0
)(0 11 −1
)=(
1 · 0 + 1 · 1 1 · 1− 1 · 11 · 0 + 1 · 0 1 · 1 + 0 · −1
)=(
1 00 1
)RemarkA matrix that is invertible is called non-singular.
16 / 102
![Page 23: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/23.jpg)
Matrix inverses
The inverse is defined as the vector A−1 such that
AA−1 = A−1A = In
Example(1 11 0
)−1=(
0 11 −1
)=⇒
(1 11 0
)(0 11 −1
)=(
1 · 0 + 1 · 1 1 · 1− 1 · 11 · 0 + 1 · 0 1 · 1 + 0 · −1
)=(
1 00 1
)RemarkA matrix that is invertible is called non-singular.
16 / 102
![Page 24: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/24.jpg)
Matrix inverses
The inverse is defined as the vector A−1 such that
AA−1 = A−1A = In
Example(1 11 0
)−1=(
0 11 −1
)=⇒
(1 11 0
)(0 11 −1
)=(
1 · 0 + 1 · 1 1 · 1− 1 · 11 · 0 + 1 · 0 1 · 1 + 0 · −1
)=(
1 00 1
)RemarkA matrix that is invertible is called non-singular.
16 / 102
![Page 25: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/25.jpg)
Properties of an inverse
Some properties are(BA)−1 = A−1B−1(A−1) T =
(AT)−1
17 / 102
![Page 26: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/26.jpg)
The Rank of A
Rank of A
A collection of vectors is x1, x2, ..., xn such thatc1x1 + c2x2 + ...+ cnxn 6= 0. The rank of a matrix is the number of linearindependent rows.
Theorem 1A square matrix has full rank if and only if it is nonsingular.
18 / 102
![Page 27: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/27.jpg)
The Rank of A
Rank of A
A collection of vectors is x1, x2, ..., xn such thatc1x1 + c2x2 + ...+ cnxn 6= 0. The rank of a matrix is the number of linearindependent rows.
Theorem 1A square matrix has full rank if and only if it is nonsingular.
18 / 102
![Page 28: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/28.jpg)
Other Theorems
A null vector x is such that Ax = 0Theorem 2: A matrix A has full column rank if and only if it does nothave a null vector.
Then, for squared matrices, we haveCorollary 3: A square matrix A is singular if and only if it has a nullvector.
19 / 102
![Page 29: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/29.jpg)
Other Theorems
A null vector x is such that Ax = 0Theorem 2: A matrix A has full column rank if and only if it does nothave a null vector.
Then, for squared matrices, we haveCorollary 3: A square matrix A is singular if and only if it has a nullvector.
19 / 102
![Page 30: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/30.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
20 / 102
![Page 31: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/31.jpg)
Determinants
A determinant can be defined recursively as follows
det(A) =
a11 if n = 1n∑j=1
(−1)1+ja1jdet(A[1j]) if n > 1 (1)
Where (−1)i+jdet(A[ij]) is called a cofactor
21 / 102
![Page 32: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/32.jpg)
Determinants
A determinant can be defined recursively as follows
det(A) =
a11 if n = 1n∑j=1
(−1)1+ja1jdet(A[1j]) if n > 1 (1)
Where (−1)i+jdet(A[ij]) is called a cofactor
21 / 102
![Page 33: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/33.jpg)
TheoremsTheorem 4(determinant properties).The determinant of a square matrix A has the following properties:
If any row or any column A is zero, then det(A) = 0.The determinant of A is multiplied by λ if the entries of any one row(or any one column) of A are all multiplied by λ.The determinant of A is unchanged if the entries in one row(respectively, column) are added to those in another row (respectively,column).The determinant of A equals the determinant of AT .The determinant of A is multiplied by −1 if any two rows (or any twocolumns) are exchanged.
Theorem 5An n× n matrix A is singular if and only if det(A) = 0.
22 / 102
![Page 34: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/34.jpg)
TheoremsTheorem 4(determinant properties).The determinant of a square matrix A has the following properties:
If any row or any column A is zero, then det(A) = 0.The determinant of A is multiplied by λ if the entries of any one row(or any one column) of A are all multiplied by λ.The determinant of A is unchanged if the entries in one row(respectively, column) are added to those in another row (respectively,column).The determinant of A equals the determinant of AT .The determinant of A is multiplied by −1 if any two rows (or any twocolumns) are exchanged.
Theorem 5An n× n matrix A is singular if and only if det(A) = 0.
22 / 102
![Page 35: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/35.jpg)
TheoremsTheorem 4(determinant properties).The determinant of a square matrix A has the following properties:
If any row or any column A is zero, then det(A) = 0.The determinant of A is multiplied by λ if the entries of any one row(or any one column) of A are all multiplied by λ.The determinant of A is unchanged if the entries in one row(respectively, column) are added to those in another row (respectively,column).The determinant of A equals the determinant of AT .The determinant of A is multiplied by −1 if any two rows (or any twocolumns) are exchanged.
Theorem 5An n× n matrix A is singular if and only if det(A) = 0.
22 / 102
![Page 36: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/36.jpg)
TheoremsTheorem 4(determinant properties).The determinant of a square matrix A has the following properties:
If any row or any column A is zero, then det(A) = 0.The determinant of A is multiplied by λ if the entries of any one row(or any one column) of A are all multiplied by λ.The determinant of A is unchanged if the entries in one row(respectively, column) are added to those in another row (respectively,column).The determinant of A equals the determinant of AT .The determinant of A is multiplied by −1 if any two rows (or any twocolumns) are exchanged.
Theorem 5An n× n matrix A is singular if and only if det(A) = 0.
22 / 102
![Page 37: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/37.jpg)
TheoremsTheorem 4(determinant properties).The determinant of a square matrix A has the following properties:
If any row or any column A is zero, then det(A) = 0.The determinant of A is multiplied by λ if the entries of any one row(or any one column) of A are all multiplied by λ.The determinant of A is unchanged if the entries in one row(respectively, column) are added to those in another row (respectively,column).The determinant of A equals the determinant of AT .The determinant of A is multiplied by −1 if any two rows (or any twocolumns) are exchanged.
Theorem 5An n× n matrix A is singular if and only if det(A) = 0.
22 / 102
![Page 38: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/38.jpg)
TheoremsTheorem 4(determinant properties).The determinant of a square matrix A has the following properties:
If any row or any column A is zero, then det(A) = 0.The determinant of A is multiplied by λ if the entries of any one row(or any one column) of A are all multiplied by λ.The determinant of A is unchanged if the entries in one row(respectively, column) are added to those in another row (respectively,column).The determinant of A equals the determinant of AT .The determinant of A is multiplied by −1 if any two rows (or any twocolumns) are exchanged.
Theorem 5An n× n matrix A is singular if and only if det(A) = 0.
22 / 102
![Page 39: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/39.jpg)
TheoremsTheorem 4(determinant properties).The determinant of a square matrix A has the following properties:
If any row or any column A is zero, then det(A) = 0.The determinant of A is multiplied by λ if the entries of any one row(or any one column) of A are all multiplied by λ.The determinant of A is unchanged if the entries in one row(respectively, column) are added to those in another row (respectively,column).The determinant of A equals the determinant of AT .The determinant of A is multiplied by −1 if any two rows (or any twocolumns) are exchanged.
Theorem 5An n× n matrix A is singular if and only if det(A) = 0.
22 / 102
![Page 40: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/40.jpg)
Positive definite matrix
DefinitionA positive definite matrix A is called positive definite if and only ifxTAx > 0 for all x 6= 0
Theorem 6For any matrix A with full column rank, the matrix ATA is positivedefinite.
23 / 102
![Page 41: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/41.jpg)
Positive definite matrix
DefinitionA positive definite matrix A is called positive definite if and only ifxTAx > 0 for all x 6= 0
Theorem 6For any matrix A with full column rank, the matrix ATA is positivedefinite.
23 / 102
![Page 42: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/42.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
24 / 102
![Page 43: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/43.jpg)
Matrix Multiplication
Problem descriptionGiven n× n matrices A,B and C:(
r st u
)=(a bc d
)(e fg h
)
Thus, you could compute r, s, t and u using recursion!!!r = ae+ bg
s = af + bh
t = ce+ dg
u = cf + dh
25 / 102
![Page 44: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/44.jpg)
Matrix Multiplication
Problem descriptionGiven n× n matrices A,B and C:(
r st u
)=(a bc d
)(e fg h
)
Thus, you could compute r, s, t and u using recursion!!!r = ae+ bg
s = af + bh
t = ce+ dg
u = cf + dh
25 / 102
![Page 45: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/45.jpg)
Problem
Complexity of previous approach
T (n) = 8T(n
2
)+ Θ(n2)
Thus
T (n) = Θ(n3)
ThereforeYou need to use a different type of products.
26 / 102
![Page 46: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/46.jpg)
Problem
Complexity of previous approach
T (n) = 8T(n
2
)+ Θ(n2)
Thus
T (n) = Θ(n3)
ThereforeYou need to use a different type of products.
26 / 102
![Page 47: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/47.jpg)
Problem
Complexity of previous approach
T (n) = 8T(n
2
)+ Θ(n2)
Thus
T (n) = Θ(n3)
ThereforeYou need to use a different type of products.
26 / 102
![Page 48: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/48.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
27 / 102
![Page 49: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/49.jpg)
The Strassen’s Algorithm
It is a divide and conquer algorithmGiven A, B, C matrices with dimensions n× n, we recursively split thematrices such that we finish with 12 n
2 ×n2 sub matrices(
r st u
)=(a bc d
)(e fg h
)
Remember the Gauss Trick?Imagine the same for Matrix Multiplication.
28 / 102
![Page 50: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/50.jpg)
The Strassen’s Algorithm
It is a divide and conquer algorithmGiven A, B, C matrices with dimensions n× n, we recursively split thematrices such that we finish with 12 n
2 ×n2 sub matrices(
r st u
)=(a bc d
)(e fg h
)
Remember the Gauss Trick?Imagine the same for Matrix Multiplication.
28 / 102
![Page 51: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/51.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
29 / 102
![Page 52: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/52.jpg)
Algorithm
Strassen’s Algorithm1 Divide the input matrices A and B into n
2 ×n2 sub matrices.
2 Using Θ(n2) scalar additions and subtractions, compute 14 matrices
A1, B1, ..., A7, B7 each of which is n2 ×
n2 .
3 Recursively compute the seven matrices products Pi = AiBi fori = 1, 2, 3, ..., 7.
4 Compute the desired matrix (r st u
)
by adding and or subtracting various combinations of the Pi matrices,using only Θ
(n2) scalar additions and subtractions
30 / 102
![Page 53: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/53.jpg)
Algorithm
Strassen’s Algorithm1 Divide the input matrices A and B into n
2 ×n2 sub matrices.
2 Using Θ(n2) scalar additions and subtractions, compute 14 matrices
A1, B1, ..., A7, B7 each of which is n2 ×
n2 .
3 Recursively compute the seven matrices products Pi = AiBi fori = 1, 2, 3, ..., 7.
4 Compute the desired matrix (r st u
)
by adding and or subtracting various combinations of the Pi matrices,using only Θ
(n2) scalar additions and subtractions
30 / 102
![Page 54: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/54.jpg)
Algorithm
Strassen’s Algorithm1 Divide the input matrices A and B into n
2 ×n2 sub matrices.
2 Using Θ(n2) scalar additions and subtractions, compute 14 matrices
A1, B1, ..., A7, B7 each of which is n2 ×
n2 .
3 Recursively compute the seven matrices products Pi = AiBi fori = 1, 2, 3, ..., 7.
4 Compute the desired matrix (r st u
)
by adding and or subtracting various combinations of the Pi matrices,using only Θ
(n2) scalar additions and subtractions
30 / 102
![Page 55: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/55.jpg)
Algorithm
Strassen’s Algorithm1 Divide the input matrices A and B into n
2 ×n2 sub matrices.
2 Using Θ(n2) scalar additions and subtractions, compute 14 matrices
A1, B1, ..., A7, B7 each of which is n2 ×
n2 .
3 Recursively compute the seven matrices products Pi = AiBi fori = 1, 2, 3, ..., 7.
4 Compute the desired matrix (r st u
)
by adding and or subtracting various combinations of the Pi matrices,using only Θ
(n2) scalar additions and subtractions
30 / 102
![Page 56: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/56.jpg)
Algorithm
Strassen’s Algorithm1 Divide the input matrices A and B into n
2 ×n2 sub matrices.
2 Using Θ(n2) scalar additions and subtractions, compute 14 matrices
A1, B1, ..., A7, B7 each of which is n2 ×
n2 .
3 Recursively compute the seven matrices products Pi = AiBi fori = 1, 2, 3, ..., 7.
4 Compute the desired matrix (r st u
)
by adding and or subtracting various combinations of the Pi matrices,using only Θ
(n2) scalar additions and subtractions
30 / 102
![Page 57: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/57.jpg)
Algorithm
Strassen’s Algorithm1 Divide the input matrices A and B into n
2 ×n2 sub matrices.
2 Using Θ(n2) scalar additions and subtractions, compute 14 matrices
A1, B1, ..., A7, B7 each of which is n2 ×
n2 .
3 Recursively compute the seven matrices products Pi = AiBi fori = 1, 2, 3, ..., 7.
4 Compute the desired matrix (r st u
)
by adding and or subtracting various combinations of the Pi matrices,using only Θ
(n2) scalar additions and subtractions
30 / 102
![Page 58: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/58.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
31 / 102
![Page 59: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/59.jpg)
Strassen Observed that
Trial and ErrorFirst , he generated
Pi = AiBi = (αi1a+ αi2b+ αi3c+ αi4d) · (βi1e+ βi2f + βi3g + βi4h)
Where αij , βij ∈ −1, 0, 1
32 / 102
![Page 60: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/60.jpg)
Then
r
r = ae+ bg =(a b c d
)+1 0 0 00 0 +1 00 0 0 00 0 0 0
efgh
s
s = af + bh =(a b c d
)+1 0 0 00 0 0 +10 0 0 00 0 0 0
efgh
33 / 102
![Page 61: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/61.jpg)
Then
r
r = ae+ bg =(a b c d
)+1 0 0 00 0 +1 00 0 0 00 0 0 0
efgh
s
s = af + bh =(a b c d
)+1 0 0 00 0 0 +10 0 0 00 0 0 0
efgh
33 / 102
![Page 62: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/62.jpg)
Then
r
r = ae+ bg =(a b c d
)+1 0 0 00 0 +1 00 0 0 00 0 0 0
efgh
s
s = af + bh =(a b c d
)+1 0 0 00 0 0 +10 0 0 00 0 0 0
efgh
33 / 102
![Page 63: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/63.jpg)
Therefore
t
r = ce+ dg =(a b c d
)0 0 0 00 0 0 0
+1 0 0 00 0 +1 0
efgh
u
u = cf + dh =(a b c d
)0 0 0 00 0 0 00 +1 0 00 0 0 +1
efgh
34 / 102
![Page 64: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/64.jpg)
Therefore
t
r = ce+ dg =(a b c d
)0 0 0 00 0 0 0
+1 0 0 00 0 +1 0
efgh
u
u = cf + dh =(a b c d
)0 0 0 00 0 0 00 +1 0 00 0 0 +1
efgh
34 / 102
![Page 65: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/65.jpg)
Example Compute the s from P1 and P2 matrices
Computes = P1 + P2
Where P1
P1 = A1B1
= a (f − h)= af − ah
=(a b c d
) 0 +1 0 −10 0 0 00 0 0 00 0 0 0
efgh
35 / 102
![Page 66: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/66.jpg)
Example Compute the s from P1 and P2 matrices
Computes = P1 + P2
Where P1
P1 = A1B1
= a (f − h)= af − ah
=(a b c d
) 0 +1 0 −10 0 0 00 0 0 00 0 0 0
efgh
35 / 102
![Page 67: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/67.jpg)
Example Compute the s from P1 and P2 matrices
Computes = P1 + P2
Where P1
P1 = A1B1
= a (f − h)= af − ah
=(a b c d
) 0 +1 0 −10 0 0 00 0 0 00 0 0 0
efgh
35 / 102
![Page 68: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/68.jpg)
Example Compute the s from P1 and P2 matrices
Computes = P1 + P2
Where P1
P1 = A1B1
= a (f − h)= af − ah
=(a b c d
) 0 +1 0 −10 0 0 00 0 0 00 0 0 0
efgh
35 / 102
![Page 69: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/69.jpg)
Example Compute the s from P1 and P2 matrices
Computes = P1 + P2
Where P1
P1 = A1B1
= a (f − h)= af − ah
=(a b c d
) 0 +1 0 −10 0 0 00 0 0 00 0 0 0
efgh
35 / 102
![Page 70: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/70.jpg)
Example Compute the s from P1 and P2 matrices
Where P2
P2 = A2B2
= (a+ b)h= ah+ bh
=(a b c d
) 0 0 0 +10 0 0 +10 0 0 00 0 0 0
efgh
36 / 102
![Page 71: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/71.jpg)
Example Compute the s from P1 and P2 matrices
Where P2
P2 = A2B2
= (a+ b)h= ah+ bh
=(a b c d
) 0 0 0 +10 0 0 +10 0 0 00 0 0 0
efgh
36 / 102
![Page 72: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/72.jpg)
Example Compute the s from P1 and P2 matrices
Where P2
P2 = A2B2
= (a+ b)h= ah+ bh
=(a b c d
) 0 0 0 +10 0 0 +10 0 0 00 0 0 0
efgh
36 / 102
![Page 73: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/73.jpg)
Example Compute the s from P1 and P2 matrices
Where P2
P2 = A2B2
= (a+ b)h= ah+ bh
=(a b c d
) 0 0 0 +10 0 0 +10 0 0 00 0 0 0
efgh
36 / 102
![Page 74: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/74.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
37 / 102
![Page 75: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/75.jpg)
Complexity
Because we are only computing 7 matricesT (n) = 7T
(n2)
+ Θ(n2) = Θ
(nlg 7
)= O
(n2.81).
38 / 102
![Page 76: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/76.jpg)
Nevertheless
We do not use Strassen’s becauseA constant factor hidden in the running of the algorithm is larger thanthe constant factor of the naive Θ
(n3) method.
When matrices are sparse, there are faster methods.Strassen’s is not a numerically stable as the naive method.The sub matrices formed at the levels of the recursion consume space.
39 / 102
![Page 77: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/77.jpg)
Nevertheless
We do not use Strassen’s becauseA constant factor hidden in the running of the algorithm is larger thanthe constant factor of the naive Θ
(n3) method.
When matrices are sparse, there are faster methods.Strassen’s is not a numerically stable as the naive method.The sub matrices formed at the levels of the recursion consume space.
39 / 102
![Page 78: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/78.jpg)
Nevertheless
We do not use Strassen’s becauseA constant factor hidden in the running of the algorithm is larger thanthe constant factor of the naive Θ
(n3) method.
When matrices are sparse, there are faster methods.Strassen’s is not a numerically stable as the naive method.The sub matrices formed at the levels of the recursion consume space.
39 / 102
![Page 79: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/79.jpg)
Nevertheless
We do not use Strassen’s becauseA constant factor hidden in the running of the algorithm is larger thanthe constant factor of the naive Θ
(n3) method.
When matrices are sparse, there are faster methods.Strassen’s is not a numerically stable as the naive method.The sub matrices formed at the levels of the recursion consume space.
39 / 102
![Page 80: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/80.jpg)
The Holy Grail of Matrix Multiplications O (n2)
In a method by Virginia Vassilevska Williams (2012) AssistantProfessor at Stanford
The computational complexity of her method is ω < 2.3727 orO(n2.3727)
Better than Coppersmith and Winograd (1990) O(n2.375477)
Many Researchers Believe thatCoppersmith, Winograd and Cohn et al. conjecture could lead toO(n2), contradicting a variant of the widely believed sun flower
conjecture of Erdos and Rado.
40 / 102
![Page 81: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/81.jpg)
The Holy Grail of Matrix Multiplications O (n2)
In a method by Virginia Vassilevska Williams (2012) AssistantProfessor at Stanford
The computational complexity of her method is ω < 2.3727 orO(n2.3727)
Better than Coppersmith and Winograd (1990) O(n2.375477)
Many Researchers Believe thatCoppersmith, Winograd and Cohn et al. conjecture could lead toO(n2), contradicting a variant of the widely believed sun flower
conjecture of Erdos and Rado.
40 / 102
![Page 82: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/82.jpg)
The Holy Grail of Matrix Multiplications O (n2)
In a method by Virginia Vassilevska Williams (2012) AssistantProfessor at Stanford
The computational complexity of her method is ω < 2.3727 orO(n2.3727)
Better than Coppersmith and Winograd (1990) O(n2.375477)
Many Researchers Believe thatCoppersmith, Winograd and Cohn et al. conjecture could lead toO(n2), contradicting a variant of the widely believed sun flower
conjecture of Erdos and Rado.
40 / 102
![Page 83: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/83.jpg)
Exercises
28.1-328.1-528.1-828.1-928.2-228.2-5
41 / 102
![Page 84: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/84.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
42 / 102
![Page 85: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/85.jpg)
In Many Fields
From Optimization to ControlWe are required to solve systems of simultaneous equations.
For ExampleFor Polynomial Curve Fitting, we are given (x1, y1) , (x2, y2) , ..., (xn, yn)
We wantTo find a polynomial of degree n− 1 with structure
p (x) = a0 + a1x+ a2x2 + ...+ an−1x
n−1
43 / 102
![Page 86: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/86.jpg)
In Many Fields
From Optimization to ControlWe are required to solve systems of simultaneous equations.
For ExampleFor Polynomial Curve Fitting, we are given (x1, y1) , (x2, y2) , ..., (xn, yn)
We wantTo find a polynomial of degree n− 1 with structure
p (x) = a0 + a1x+ a2x2 + ...+ an−1x
n−1
43 / 102
![Page 87: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/87.jpg)
In Many Fields
From Optimization to ControlWe are required to solve systems of simultaneous equations.
For ExampleFor Polynomial Curve Fitting, we are given (x1, y1) , (x2, y2) , ..., (xn, yn)
We wantTo find a polynomial of degree n− 1 with structure
p (x) = a0 + a1x+ a2x2 + ...+ an−1x
n−1
43 / 102
![Page 88: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/88.jpg)
Thus
We can build a system of equations
a0 + a1x1 + a2x21 + ...+ an−1x
n−11 = y1
a0 + a1x2 + a2x22 + ...+ an−1x
n−12 = y2
...a0 + a1xn + a2x
2n + ...+ an−1x
n−1n = yn
We have n unknowns
a0, a1, a2, ..., an−1
44 / 102
![Page 89: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/89.jpg)
Thus
We can build a system of equations
a0 + a1x1 + a2x21 + ...+ an−1x
n−11 = y1
a0 + a1x2 + a2x22 + ...+ an−1x
n−12 = y2
...a0 + a1xn + a2x
2n + ...+ an−1x
n−1n = yn
We have n unknowns
a0, a1, a2, ..., an−1
44 / 102
![Page 90: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/90.jpg)
Solving Systems of Linear Equations
Proceed as followsWe start with a set of linear equations with n unknowns:
x1, x2, ..., xn
a11x1 + a12x2 + ...+ a1nxn = b1
a21x1 + a22x2 + ...+ a2nxn = b2...
...an1x1 + an2x2 + ...+ annxn = bn
Something NotableA set of values for x1, x2, ..., xn that satisfy all of the equationssimultaneously is said to be a solution to these equations.In this section, we only treat the case in which there are exactly nequations in n unknowns.
45 / 102
![Page 91: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/91.jpg)
Solving Systems of Linear Equations
Proceed as followsWe start with a set of linear equations with n unknowns:
x1, x2, ..., xn
a11x1 + a12x2 + ...+ a1nxn = b1
a21x1 + a22x2 + ...+ a2nxn = b2...
...an1x1 + an2x2 + ...+ annxn = bn
Something NotableA set of values for x1, x2, ..., xn that satisfy all of the equationssimultaneously is said to be a solution to these equations.In this section, we only treat the case in which there are exactly nequations in n unknowns.
45 / 102
![Page 92: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/92.jpg)
Solving Systems of Linear Equations
Proceed as followsWe start with a set of linear equations with n unknowns:
x1, x2, ..., xn
a11x1 + a12x2 + ...+ a1nxn = b1
a21x1 + a22x2 + ...+ a2nxn = b2...
...an1x1 + an2x2 + ...+ annxn = bn
Something NotableA set of values for x1, x2, ..., xn that satisfy all of the equationssimultaneously is said to be a solution to these equations.In this section, we only treat the case in which there are exactly nequations in n unknowns.
45 / 102
![Page 93: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/93.jpg)
Solving Systems of Linear Equations
Proceed as followsWe start with a set of linear equations with n unknowns:
x1, x2, ..., xn
a11x1 + a12x2 + ...+ a1nxn = b1
a21x1 + a22x2 + ...+ a2nxn = b2...
...an1x1 + an2x2 + ...+ annxn = bn
Something NotableA set of values for x1, x2, ..., xn that satisfy all of the equationssimultaneously is said to be a solution to these equations.In this section, we only treat the case in which there are exactly nequations in n unknowns.
45 / 102
![Page 94: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/94.jpg)
Solving systems of linear equations
continuationWe can conveniently rewrite the equations as the matrix-vectorequation:
a11 a12 . . . a1na21 a22 . . . a2n...
... . . . ...an1 an2 . . . ann
x1x2...xn
=
b1b2...bn
or, equivalently, letting A = (aij), x = (xj), and b = (bi), as
Ax = b
In this section, we shall be concerned predominantly with the case ofwhich A is nonsingular, after all we want to invert A.
46 / 102
![Page 95: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/95.jpg)
Solving systems of linear equations
continuationWe can conveniently rewrite the equations as the matrix-vectorequation:
a11 a12 . . . a1na21 a22 . . . a2n...
... . . . ...an1 an2 . . . ann
x1x2...xn
=
b1b2...bn
or, equivalently, letting A = (aij), x = (xj), and b = (bi), as
Ax = b
In this section, we shall be concerned predominantly with the case ofwhich A is nonsingular, after all we want to invert A.
46 / 102
![Page 96: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/96.jpg)
Solving systems of linear equations
continuationWe can conveniently rewrite the equations as the matrix-vectorequation:
a11 a12 . . . a1na21 a22 . . . a2n...
... . . . ...an1 an2 . . . ann
x1x2...xn
=
b1b2...bn
or, equivalently, letting A = (aij), x = (xj), and b = (bi), as
Ax = b
In this section, we shall be concerned predominantly with the case ofwhich A is nonsingular, after all we want to invert A.
46 / 102
![Page 97: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/97.jpg)
Solving systems of linear equations
continuationWe can conveniently rewrite the equations as the matrix-vectorequation:
a11 a12 . . . a1na21 a22 . . . a2n...
... . . . ...an1 an2 . . . ann
x1x2...xn
=
b1b2...bn
or, equivalently, letting A = (aij), x = (xj), and b = (bi), as
Ax = b
In this section, we shall be concerned predominantly with the case ofwhich A is nonsingular, after all we want to invert A.
46 / 102
![Page 98: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/98.jpg)
Solving systems of linear equations
continuationWe can conveniently rewrite the equations as the matrix-vectorequation:
a11 a12 . . . a1na21 a22 . . . a2n...
... . . . ...an1 an2 . . . ann
x1x2...xn
=
b1b2...bn
or, equivalently, letting A = (aij), x = (xj), and b = (bi), as
Ax = b
In this section, we shall be concerned predominantly with the case ofwhich A is nonsingular, after all we want to invert A.
46 / 102
![Page 99: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/99.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
47 / 102
![Page 100: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/100.jpg)
Overview of Lower Upper (LUP) Decomposition
IntuitionThe idea behind LUP decomposition is to find three n× n matrices L,U ,and P such that:
PA = LU
where:L is a unit lower triangular matrix.U is an upper triangular matrix.P is a permutation matrix.
WhereWe call matrices L,U , and P satisfying the above equation a LUPdecomposition of the matrix A.
48 / 102
![Page 101: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/101.jpg)
Overview of Lower Upper (LUP) Decomposition
IntuitionThe idea behind LUP decomposition is to find three n× n matrices L,U ,and P such that:
PA = LU
where:L is a unit lower triangular matrix.U is an upper triangular matrix.P is a permutation matrix.
WhereWe call matrices L,U , and P satisfying the above equation a LUPdecomposition of the matrix A.
48 / 102
![Page 102: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/102.jpg)
Overview of Lower Upper (LUP) Decomposition
IntuitionThe idea behind LUP decomposition is to find three n× n matrices L,U ,and P such that:
PA = LU
where:L is a unit lower triangular matrix.U is an upper triangular matrix.P is a permutation matrix.
WhereWe call matrices L,U , and P satisfying the above equation a LUPdecomposition of the matrix A.
48 / 102
![Page 103: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/103.jpg)
Overview of Lower Upper (LUP) Decomposition
IntuitionThe idea behind LUP decomposition is to find three n× n matrices L,U ,and P such that:
PA = LU
where:L is a unit lower triangular matrix.U is an upper triangular matrix.P is a permutation matrix.
WhereWe call matrices L,U , and P satisfying the above equation a LUPdecomposition of the matrix A.
48 / 102
![Page 104: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/104.jpg)
Overview of Lower Upper (LUP) Decomposition
IntuitionThe idea behind LUP decomposition is to find three n× n matrices L,U ,and P such that:
PA = LU
where:L is a unit lower triangular matrix.U is an upper triangular matrix.P is a permutation matrix.
WhereWe call matrices L,U , and P satisfying the above equation a LUPdecomposition of the matrix A.
48 / 102
![Page 105: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/105.jpg)
Overview of Lower Upper (LUP) Decomposition
IntuitionThe idea behind LUP decomposition is to find three n× n matrices L,U ,and P such that:
PA = LU
where:L is a unit lower triangular matrix.U is an upper triangular matrix.P is a permutation matrix.
WhereWe call matrices L,U , and P satisfying the above equation a LUPdecomposition of the matrix A.
48 / 102
![Page 106: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/106.jpg)
Overview of Lower Upper (LUP) Decomposition
IntuitionThe idea behind LUP decomposition is to find three n× n matrices L,U ,and P such that:
PA = LU
where:L is a unit lower triangular matrix.U is an upper triangular matrix.P is a permutation matrix.
WhereWe call matrices L,U , and P satisfying the above equation a LUPdecomposition of the matrix A.
48 / 102
![Page 107: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/107.jpg)
What is a Permutation Matrix
BasicallyWe represent the permutation P compactly by an array π[1..n]. Fori = 1, 2, ..., n, the entry π[i] indicates that Piπ[i] = 1 and Pij = 0 forj 6= π[i].
ThusPA has aπ[i],j in row i and a column j.Pb has bπ[i] as its ith element.
49 / 102
![Page 108: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/108.jpg)
What is a Permutation Matrix
BasicallyWe represent the permutation P compactly by an array π[1..n]. Fori = 1, 2, ..., n, the entry π[i] indicates that Piπ[i] = 1 and Pij = 0 forj 6= π[i].
ThusPA has aπ[i],j in row i and a column j.Pb has bπ[i] as its ith element.
49 / 102
![Page 109: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/109.jpg)
How can we use this in our advantage?
Lock at this
Ax = b =⇒ PAx = Pb (2)
Therefore
LUx = Pb (3)
Now, if we make Ux = y
Ly = Pb (4)
50 / 102
![Page 110: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/110.jpg)
How can we use this in our advantage?
Lock at this
Ax = b =⇒ PAx = Pb (2)
Therefore
LUx = Pb (3)
Now, if we make Ux = y
Ly = Pb (4)
50 / 102
![Page 111: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/111.jpg)
How can we use this in our advantage?
Lock at this
Ax = b =⇒ PAx = Pb (2)
Therefore
LUx = Pb (3)
Now, if we make Ux = y
Ly = Pb (4)
50 / 102
![Page 112: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/112.jpg)
Thus
We first obtain y
Then, we obtain x.
51 / 102
![Page 113: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/113.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
52 / 102
![Page 114: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/114.jpg)
Forward and Back Substitution
Forward substitutionForward substitution can solve the lower triangular system Ly = Pb inΘ(n2) time, given L, P and b.
ThenSince L is unit lower triangular, equation Ly = Pb can be rewritten as:
y1 = bπ[1]
l21y1 + y2 = bπ[2]
l31y1 + l32 + y3 = bπ[3]...
ln1y1 + ln2y2 + ln3y3 + ...+ yn = bπ[n]
53 / 102
![Page 115: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/115.jpg)
Forward and Back Substitution
Forward substitutionForward substitution can solve the lower triangular system Ly = Pb inΘ(n2) time, given L, P and b.
ThenSince L is unit lower triangular, equation Ly = Pb can be rewritten as:
y1 = bπ[1]
l21y1 + y2 = bπ[2]
l31y1 + l32 + y3 = bπ[3]...
ln1y1 + ln2y2 + ln3y3 + ...+ yn = bπ[n]
53 / 102
![Page 116: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/116.jpg)
Forward and Back Substitution
Back substitutionBack substitution is similar to forward substitution. Like forwardsubstitution, this process runs in Θ(n2) time. Since U is upper-triangular,we can rewrite the system Ux = y as
u11x1 + u12x2 + ...+ u1n−2xn−2 + u1n−1xn−1 + u1nxn = y1
u22x2 + ...+ u2n−2xn−2 + u2n−1xn−1 + u2nxn = y2...
un−2n−2xn−2 + un−2n−1xn−1 + un−2nxn = yn−2
un−1n−1xn−1 + un−1nxn = yn−1
unnxn = yn
54 / 102
![Page 117: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/117.jpg)
Example
We have
Ax =
1 2 03 4 45 6 3
x =
378
= b
55 / 102
![Page 118: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/118.jpg)
Example
The L, U and P matrix
L =
1 0 00.2 1 00.6 0.5 1
, U =
5 6 30 0.8 −0.60 0 2.5
, P =
0 0 11 0 00 1 0
56 / 102
![Page 119: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/119.jpg)
Example
Using forward substitution, Ly = Pb for y
Ly =
1 0 00.2 1 00.6 0.5 1
y =
0 0 11 0 00 1 0
3
78
= Pb
57 / 102
![Page 120: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/120.jpg)
Example
Using forward substitution, we get y
y =
81.41.5
58 / 102
![Page 121: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/121.jpg)
Example
Now, we use the back substitution, Ux = y for x
Ux =
5 6 30 0.8 −0.60 0 2.5
x1x2x3
=
81.41.5
,
59 / 102
![Page 122: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/122.jpg)
Example
Finally, we get
x =
−1.42.20.6
60 / 102
![Page 123: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/123.jpg)
Forward and Back SubstitutionGiven P , L, U , and b, the procedure LUP- SOLVE solves for x bycombining forward and back substitutionLUP-SOLVE(L,U, π, b)
1 n = L.rows
2 Let x be a new vector of length n3 for i = 1 to n
4 yi = bπ[i] −∑i−1j=1 lijyj
5 for i = n downto 1
6 xi =
(yi−∑n
j=i+1 uijxj
)uii
7 return x
ComplexityThe running time is Θ(n2).
61 / 102
![Page 124: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/124.jpg)
Forward and Back SubstitutionGiven P , L, U , and b, the procedure LUP- SOLVE solves for x bycombining forward and back substitutionLUP-SOLVE(L,U, π, b)
1 n = L.rows
2 Let x be a new vector of length n3 for i = 1 to n
4 yi = bπ[i] −∑i−1j=1 lijyj
5 for i = n downto 1
6 xi =
(yi−∑n
j=i+1 uijxj
)uii
7 return x
ComplexityThe running time is Θ(n2).
61 / 102
![Page 125: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/125.jpg)
Forward and Back SubstitutionGiven P , L, U , and b, the procedure LUP- SOLVE solves for x bycombining forward and back substitutionLUP-SOLVE(L,U, π, b)
1 n = L.rows
2 Let x be a new vector of length n3 for i = 1 to n
4 yi = bπ[i] −∑i−1j=1 lijyj
5 for i = n downto 1
6 xi =
(yi−∑n
j=i+1 uijxj
)uii
7 return x
ComplexityThe running time is Θ(n2).
61 / 102
![Page 126: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/126.jpg)
Forward and Back SubstitutionGiven P , L, U , and b, the procedure LUP- SOLVE solves for x bycombining forward and back substitutionLUP-SOLVE(L,U, π, b)
1 n = L.rows
2 Let x be a new vector of length n3 for i = 1 to n
4 yi = bπ[i] −∑i−1j=1 lijyj
5 for i = n downto 1
6 xi =
(yi−∑n
j=i+1 uijxj
)uii
7 return x
ComplexityThe running time is Θ(n2).
61 / 102
![Page 127: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/127.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
62 / 102
![Page 128: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/128.jpg)
Ok, if we have the L,U and P !!!
ThusWe need to find those matrices
How, we do it?We are going to use something called the Gaussian Elimination.
63 / 102
![Page 129: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/129.jpg)
Ok, if we have the L,U and P !!!
ThusWe need to find those matrices
How, we do it?We are going to use something called the Gaussian Elimination.
63 / 102
![Page 130: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/130.jpg)
For this
We assume that A is a n× n
Such that A is not singular
We use a process known as Gaussian elimination to create LUdecompositionThis algorithm is recursive in nature.
PropertiesClearly if n = 1, we are done for L = I1 and U = A.
64 / 102
![Page 131: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/131.jpg)
For this
We assume that A is a n× n
Such that A is not singular
We use a process known as Gaussian elimination to create LUdecompositionThis algorithm is recursive in nature.
PropertiesClearly if n = 1, we are done for L = I1 and U = A.
64 / 102
![Page 132: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/132.jpg)
For this
We assume that A is a n× n
Such that A is not singular
We use a process known as Gaussian elimination to create LUdecompositionThis algorithm is recursive in nature.
PropertiesClearly if n = 1, we are done for L = I1 and U = A.
64 / 102
![Page 133: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/133.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
65 / 102
![Page 134: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/134.jpg)
Computing LU decomposition
For n > 1, we break A into four parts
A =
a11 a12 · · · a1n
a21 a22 · · · a2n...
... . . . ...an1 an2 · · · ann
=
a11 wT
v A′
(5)
66 / 102
![Page 135: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/135.jpg)
Where
We havev is a column (n− 1)−vector.wT is a row (n− 1)−vector.A′ is an (n− 1)× (n− 1).
67 / 102
![Page 136: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/136.jpg)
Where
We havev is a column (n− 1)−vector.wT is a row (n− 1)−vector.A′ is an (n− 1)× (n− 1).
67 / 102
![Page 137: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/137.jpg)
Where
We havev is a column (n− 1)−vector.wT is a row (n− 1)−vector.A′ is an (n− 1)× (n− 1).
67 / 102
![Page 138: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/138.jpg)
Where
We havev is a column (n− 1)−vector.wT is a row (n− 1)−vector.A′ is an (n− 1)× (n− 1).
67 / 102
![Page 139: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/139.jpg)
Computing a LU decomposition
Thus, we can do the following
A =(a11 wT
v A′
)
=(
1 0va11
In−1
)a11 wT
0 A′ − vwT
a11︸ ︷︷ ︸Schur Complement
=(
1 0va11
In−1
)(a11 wT
0 L′U ′
)
=(
1 0va11
L′
)(a11 wT
0 U ′
)= LU
68 / 102
![Page 140: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/140.jpg)
Computing a LU decomposition
Thus, we can do the following
A =(a11 wT
v A′
)
=(
1 0va11
In−1
)a11 wT
0 A′ − vwT
a11︸ ︷︷ ︸Schur Complement
=(
1 0va11
In−1
)(a11 wT
0 L′U ′
)
=(
1 0va11
L′
)(a11 wT
0 U ′
)= LU
68 / 102
![Page 141: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/141.jpg)
Computing a LU decomposition
Thus, we can do the following
A =(a11 wT
v A′
)
=(
1 0va11
In−1
)a11 wT
0 A′ − vwT
a11︸ ︷︷ ︸Schur Complement
=(
1 0va11
In−1
)(a11 wT
0 L′U ′
)
=(
1 0va11
L′
)(a11 wT
0 U ′
)= LU
68 / 102
![Page 142: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/142.jpg)
Computing a LU decomposition
Thus, we can do the following
A =(a11 wT
v A′
)
=(
1 0va11
In−1
)a11 wT
0 A′ − vwT
a11︸ ︷︷ ︸Schur Complement
=(
1 0va11
In−1
)(a11 wT
0 L′U ′
)
=(
1 0va11
L′
)(a11 wT
0 U ′
)= LU
68 / 102
![Page 143: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/143.jpg)
Computing a LU decomposition
Thus, we can do the following
A =(a11 wT
v A′
)
=(
1 0va11
In−1
)a11 wT
0 A′ − vwT
a11︸ ︷︷ ︸Schur Complement
=(
1 0va11
In−1
)(a11 wT
0 L′U ′
)
=(
1 0va11
L′
)(a11 wT
0 U ′
)= LU
68 / 102
![Page 144: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/144.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 145: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/145.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 146: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/146.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 147: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/147.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 148: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/148.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 149: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/149.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 150: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/150.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 151: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/151.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 152: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/152.jpg)
Computing a LU decompositionPseudo-Code running in Θ (n3)LU-Decomposition(A)
1 n = A.rows
2 Let L and U be new n× n matrices3 Initialize U with 0’s below the diagonal4 Initialize L with 1’s on the diagonal and 0’s above the diagonal.5 for k = 1 to n
6 ukk = akk
7 for i = k + 1 to n
8 lik = aikukk
/ lik holds vi
9 uki = aki / uki holds wTi
10 for i = k + 1 to n
11 for j = k + 1 to n
12 aij = aij − likukj
13 return L and U
69 / 102
![Page 153: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/153.jpg)
ExampleHere, we have this example
2 3 1 56 13 5 192 19 10 234 10 11 31
⇒
( 13 5 1919 10 2310 11 31
)− 1
2
( 624
)(3 1 5
)=
( 13 5 1919 10 2310 11 31
)− 1
2
( 18 6 306 2 1012 4 20
)⇒
2 3 1 53 4 2 41 16 9 182 4 9 21
⇒(
9 189 11
)− 1
4
(164
)(2 4
)=(
9 189 11
)− 1
4
(32 648 16
)=
(9 189 11
)−(
8 162 4
)⇒
2 3 1 53 4 2 41 4 1 22 1 7 17
⇒
2 3 1 53 4 2 41 4 1 22 1 7 3
70 / 102
![Page 154: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/154.jpg)
ExampleHere, we have this example
2 3 1 56 13 5 192 19 10 234 10 11 31
⇒
( 13 5 1919 10 2310 11 31
)− 1
2
( 624
)(3 1 5
)=
( 13 5 1919 10 2310 11 31
)− 1
2
( 18 6 306 2 1012 4 20
)⇒
2 3 1 53 4 2 41 16 9 182 4 9 21
⇒(
9 189 11
)− 1
4
(164
)(2 4
)=(
9 189 11
)− 1
4
(32 648 16
)=
(9 189 11
)−(
8 162 4
)⇒
2 3 1 53 4 2 41 4 1 22 1 7 17
⇒
2 3 1 53 4 2 41 4 1 22 1 7 3
70 / 102
![Page 155: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/155.jpg)
ExampleHere, we have this example
2 3 1 56 13 5 192 19 10 234 10 11 31
⇒
( 13 5 1919 10 2310 11 31
)− 1
2
( 624
)(3 1 5
)=
( 13 5 1919 10 2310 11 31
)− 1
2
( 18 6 306 2 1012 4 20
)⇒
2 3 1 53 4 2 41 16 9 182 4 9 21
⇒(
9 189 11
)− 1
4
(164
)(2 4
)=(
9 189 11
)− 1
4
(32 648 16
)=
(9 189 11
)−(
8 162 4
)⇒
2 3 1 53 4 2 41 4 1 22 1 7 17
⇒
2 3 1 53 4 2 41 4 1 22 1 7 3
70 / 102
![Page 156: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/156.jpg)
ExampleHere, we have this example
2 3 1 56 13 5 192 19 10 234 10 11 31
⇒
( 13 5 1919 10 2310 11 31
)− 1
2
( 624
)(3 1 5
)=
( 13 5 1919 10 2310 11 31
)− 1
2
( 18 6 306 2 1012 4 20
)⇒
2 3 1 53 4 2 41 16 9 182 4 9 21
⇒(
9 189 11
)− 1
4
(164
)(2 4
)=(
9 189 11
)− 1
4
(32 648 16
)=
(9 189 11
)−(
8 162 4
)⇒
2 3 1 53 4 2 41 4 1 22 1 7 17
⇒
2 3 1 53 4 2 41 4 1 22 1 7 3
70 / 102
![Page 157: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/157.jpg)
ExampleHere, we have this example
2 3 1 56 13 5 192 19 10 234 10 11 31
⇒
( 13 5 1919 10 2310 11 31
)− 1
2
( 624
)(3 1 5
)=
( 13 5 1919 10 2310 11 31
)− 1
2
( 18 6 306 2 1012 4 20
)⇒
2 3 1 53 4 2 41 16 9 182 4 9 21
⇒(
9 189 11
)− 1
4
(164
)(2 4
)=(
9 189 11
)− 1
4
(32 648 16
)=
(9 189 11
)−(
8 162 4
)⇒
2 3 1 53 4 2 41 4 1 22 1 7 17
⇒
2 3 1 53 4 2 41 4 1 22 1 7 3
70 / 102
![Page 158: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/158.jpg)
ExampleHere, we have this example
2 3 1 56 13 5 192 19 10 234 10 11 31
⇒
( 13 5 1919 10 2310 11 31
)− 1
2
( 624
)(3 1 5
)=
( 13 5 1919 10 2310 11 31
)− 1
2
( 18 6 306 2 1012 4 20
)⇒
2 3 1 53 4 2 41 16 9 182 4 9 21
⇒(
9 189 11
)− 1
4
(164
)(2 4
)=(
9 189 11
)− 1
4
(32 648 16
)=
(9 189 11
)−(
8 162 4
)⇒
2 3 1 53 4 2 41 4 1 22 1 7 17
⇒
2 3 1 53 4 2 41 4 1 22 1 7 3
70 / 102
![Page 159: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/159.jpg)
Thus
We get the following decomposition2 3 1 56 13 5 192 19 10 234 10 11 31
=
1 0 0 03 1 0 01 4 1 02 1 7 1
2 3 1 50 4 2 40 0 1 20 0 0 3
71 / 102
![Page 160: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/160.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
72 / 102
![Page 161: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/161.jpg)
Observations
Something NotableThe elements by which we divide during LU decomposition are calledpivots.They occupy the diagonal elements of the matrix U .
Why the permutation P
It allows us to avoid dividing by 0.
73 / 102
![Page 162: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/162.jpg)
Observations
Something NotableThe elements by which we divide during LU decomposition are calledpivots.They occupy the diagonal elements of the matrix U .
Why the permutation P
It allows us to avoid dividing by 0.
73 / 102
![Page 163: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/163.jpg)
Observations
Something NotableThe elements by which we divide during LU decomposition are calledpivots.They occupy the diagonal elements of the matrix U .
Why the permutation P
It allows us to avoid dividing by 0.
73 / 102
![Page 164: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/164.jpg)
Thus, What do we want?
We want P , L and U
PA = LU
However, we move a non-zero element, ak1
From somewhere in the first column to the (1, 1) position of the matrix.
In additionak1 as the element in the first column with the greatest absolute value.
74 / 102
![Page 165: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/165.jpg)
Thus, What do we want?
We want P , L and U
PA = LU
However, we move a non-zero element, ak1
From somewhere in the first column to the (1, 1) position of the matrix.
In additionak1 as the element in the first column with the greatest absolute value.
74 / 102
![Page 166: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/166.jpg)
Thus, What do we want?
We want P , L and U
PA = LU
However, we move a non-zero element, ak1
From somewhere in the first column to the (1, 1) position of the matrix.
In additionak1 as the element in the first column with the greatest absolute value.
74 / 102
![Page 167: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/167.jpg)
Exchange Rows
ThusWe exchange row 1 with row k, or multiplying A by a permutation matrixQ on the left
QA =(ak1 wT
v A′
)
Withv = (a21, a31, ..., an1)T with a11 replaces ak1.wT = (ak2, ak3, ..., akn).A′ is a (n− 1)× (n− 1)
75 / 102
![Page 168: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/168.jpg)
Exchange Rows
ThusWe exchange row 1 with row k, or multiplying A by a permutation matrixQ on the left
QA =(ak1 wT
v A′
)
Withv = (a21, a31, ..., an1)T with a11 replaces ak1.wT = (ak2, ak3, ..., akn).A′ is a (n− 1)× (n− 1)
75 / 102
![Page 169: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/169.jpg)
Now, ak1 6= 0
We have then
QA =(ak1 wT
v A′
)
=(
1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
76 / 102
![Page 170: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/170.jpg)
Now, ak1 6= 0
We have then
QA =(ak1 wT
v A′
)
=(
1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
76 / 102
![Page 171: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/171.jpg)
Important
Something Notableif A is nonsingular, then the Schur complement A′ − vwT
ak1is nonsingular,
too.
Now, we can find recursively an LUP decomposition for it
P ′(A′ − vwT
ak1
)= L′U ′
Then, we define a new permutation matrix
P =(
1 00 P ′
)Q
77 / 102
![Page 172: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/172.jpg)
Important
Something Notableif A is nonsingular, then the Schur complement A′ − vwT
ak1is nonsingular,
too.
Now, we can find recursively an LUP decomposition for it
P ′(A′ − vwT
ak1
)= L′U ′
Then, we define a new permutation matrix
P =(
1 00 P ′
)Q
77 / 102
![Page 173: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/173.jpg)
Important
Something Notableif A is nonsingular, then the Schur complement A′ − vwT
ak1is nonsingular,
too.
Now, we can find recursively an LUP decomposition for it
P ′(A′ − vwT
ak1
)= L′U ′
Then, we define a new permutation matrix
P =(
1 00 P ′
)Q
77 / 102
![Page 174: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/174.jpg)
ThusWe have
PA =(
1 00 P ′
)QA
=(
1 00 P ′
)(1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
P ′
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 P ′(A′ − vwT
ak1
) )
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 L′U ′
)=(
1 0P ′ vak1
L′
)(ak1 wT
0 U ′
)= LU
78 / 102
![Page 175: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/175.jpg)
ThusWe have
PA =(
1 00 P ′
)QA
=(
1 00 P ′
)(1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
P ′
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 P ′(A′ − vwT
ak1
) )
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 L′U ′
)=(
1 0P ′ vak1
L′
)(ak1 wT
0 U ′
)= LU
78 / 102
![Page 176: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/176.jpg)
ThusWe have
PA =(
1 00 P ′
)QA
=(
1 00 P ′
)(1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
P ′
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 P ′(A′ − vwT
ak1
) )
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 L′U ′
)=(
1 0P ′ vak1
L′
)(ak1 wT
0 U ′
)= LU
78 / 102
![Page 177: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/177.jpg)
ThusWe have
PA =(
1 00 P ′
)QA
=(
1 00 P ′
)(1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
P ′
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 P ′(A′ − vwT
ak1
) )
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 L′U ′
)=(
1 0P ′ vak1
L′
)(ak1 wT
0 U ′
)= LU
78 / 102
![Page 178: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/178.jpg)
ThusWe have
PA =(
1 00 P ′
)QA
=(
1 00 P ′
)(1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
P ′
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 P ′(A′ − vwT
ak1
) )
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 L′U ′
)=(
1 0P ′ vak1
L′
)(ak1 wT
0 U ′
)= LU
78 / 102
![Page 179: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/179.jpg)
ThusWe have
PA =(
1 00 P ′
)QA
=(
1 00 P ′
)(1 0vak1
In−1
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
P ′
)(ak1 wT
0 A′ − vwT
ak1
)
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 P ′(A′ − vwT
ak1
) )
=(
1 0P ′ vak1
In−1
)(ak1 wT
0 L′U ′
)=(
1 0P ′ vak1
L′
)(ak1 wT
0 U ′
)= LU
78 / 102
![Page 180: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/180.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 181: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/181.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 182: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/182.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 183: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/183.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 184: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/184.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 185: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/185.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 186: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/186.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 187: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/187.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 188: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/188.jpg)
Computing a LUP decomposition
AlgorithmLUP-Decomposition(A)1. n = A.rows
2. Let π [1..n] new array3. for i = 1 to n
4. π [i] = i
5. for k = 1 to n
6. p = 07. for i = k to n
8. if |aik| > p
9.p = |aik|
10. k′ = i
11. if p == 012. error “Singular Matrix”13. Exchange π [k]←→ π [k′]14. for i = 1 to n
15. Exchange aki ←→ ak′i
16. for i = k + 1 to n
17. aik = aikakk
18. for j = k + 1 to n
19. aij = aij − aikakj
79 / 102
![Page 189: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/189.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 190: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/190.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 191: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/191.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 192: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/192.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 193: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/193.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 194: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/194.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 195: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/195.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 196: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/196.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 197: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/197.jpg)
Computing a LUP decompositionExample
1 2 0 2 0.6
2 3 3 4 -2
3 5 5 4 2
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 3 3 4 -2
1 2 0 2 0.6
4 -1 -2 3.4 -1
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
2 0.6 0 1.6 -3.2
1 0.4 -2 0.4 -0.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 -1 4.2 -0.6
=⇒
3 5 5 4 2
1 0.4 -2 0.4 -0.2
2 0.6 0 1.6 -3.2
4 -1 0.5 4 -0.580 / 102
![Page 198: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/198.jpg)
Finally, you get
The Permutation and Decomposition0 0 1 01 0 0 00 0 0 10 1 0 0
︸ ︷︷ ︸
P
2 0 2 0.63 3 4 −25 5 4 2−1 −2 3.4 −1
︸ ︷︷ ︸
A
= ...
1 0 0 0
0.4 1 0 0−0.2 0.5 1 00.6 0 0.4 1
︸ ︷︷ ︸
L
5 5 4 20 −2 0.4 −0.20 0 4 −0.50 0 0 −3
︸ ︷︷ ︸
U
81 / 102
![Page 199: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/199.jpg)
Symmetric positive-definite matrices
Lemma 28.9Any symmetric positive-definite matrix is nonsingular.
Lemma 28.10If A is a symmetric positive-definite matrix, then every leading submatrixof A is symmetric and positive-definite.
82 / 102
![Page 200: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/200.jpg)
Symmetric positive-definite matrices
Lemma 28.9Any symmetric positive-definite matrix is nonsingular.
Lemma 28.10If A is a symmetric positive-definite matrix, then every leading submatrixof A is symmetric and positive-definite.
82 / 102
![Page 201: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/201.jpg)
Symmetric positive-definite matrices
Definition: Schur complementLet A be a symmetric positive-definite matrix, and let Ak be a leadingk × k submatrix of A. Partition A as:
A =(Ak BT
B C
)
Then, the Schur complement of A with respect to Ak is defined to be
S = C −BA−1k BT
83 / 102
![Page 202: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/202.jpg)
Symmetric positive-definite matrices
Definition: Schur complementLet A be a symmetric positive-definite matrix, and let Ak be a leadingk × k submatrix of A. Partition A as:
A =(Ak BT
B C
)
Then, the Schur complement of A with respect to Ak is defined to be
S = C −BA−1k BT
83 / 102
![Page 203: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/203.jpg)
Symmetric positive-definite matrices
Definition: Schur complementLet A be a symmetric positive-definite matrix, and let Ak be a leadingk × k submatrix of A. Partition A as:
A =(Ak BT
B C
)
Then, the Schur complement of A with respect to Ak is defined to be
S = C −BA−1k BT
83 / 102
![Page 204: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/204.jpg)
Symmetric positive-definite matrices
Definition: Schur complementLet A be a symmetric positive-definite matrix, and let Ak be a leadingk × k submatrix of A. Partition A as:
A =(Ak BT
B C
)
Then, the Schur complement of A with respect to Ak is defined to be
S = C −BA−1k BT
83 / 102
![Page 205: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/205.jpg)
Symmetric positive-definite matrices
Lemma 28.11 (Schur complement lemma)If A is a symmetric positive-definite matrix and Ak is a leading k × ksubmatrix of A, then the Schur complement of A with respect to Ak issymmetric and positive-definite.
Corollary 28.12LU decomposition of a symmetric positive-definite matrix never causes adivision by 0.
84 / 102
![Page 206: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/206.jpg)
Symmetric positive-definite matrices
Lemma 28.11 (Schur complement lemma)If A is a symmetric positive-definite matrix and Ak is a leading k × ksubmatrix of A, then the Schur complement of A with respect to Ak issymmetric and positive-definite.
Corollary 28.12LU decomposition of a symmetric positive-definite matrix never causes adivision by 0.
84 / 102
![Page 207: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/207.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
85 / 102
![Page 208: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/208.jpg)
Inverting matrices
LUP decomposition can be used to compute a matrix inverseThe computation of a matrix inverse can be speed up using techniquessuch as Strassen’s algorithm for matrix multiplication.
86 / 102
![Page 209: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/209.jpg)
Computing a matrix inverse from a LUP decomposition
Proceed as followsThe equation AX = In can be viewed as a set of n distinct equationsof the form Axi = ei, for i = 1, ..., n.We have a LUP decomposition of a matrix A in the form of threematrices L,U , and P such that PA = LU .Then we use the backward-forward to solve AXi = ei.
87 / 102
![Page 210: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/210.jpg)
Complexity
FirstWe can compute each Xi in time Θ
(n2).
Thus, X can be computed in time Θ(n3).
LUP decomposition is computed in time Θ(n3).
FinallyWe can compute A−1 of a matrix A in time Θ
(n3).
88 / 102
![Page 211: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/211.jpg)
Complexity
FirstWe can compute each Xi in time Θ
(n2).
Thus, X can be computed in time Θ(n3).
LUP decomposition is computed in time Θ(n3).
FinallyWe can compute A−1 of a matrix A in time Θ
(n3).
88 / 102
![Page 212: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/212.jpg)
Matrix multiplication and matrix inversion
Theorem 28.7If we can invert an n× n matrix in time I(n), where I(n) = Ω(n2) andI(n) satisfies the regularity condition I(3n) = O(I(n)), then we canmultiply two n× n matrices in time O(I(n)).
89 / 102
![Page 213: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/213.jpg)
Matrix multiplication and matrix inversion
Theorem 28.8If we can multiply two n× n real matrices in time M(n), whereM(n) = Ω(n2) and M(n) = O(M(n+ k)) for any k in range 0 ≤ k ≤ nand M(n2 ) ≤ cM(n) for some constant c < 1
2 . Then we can compute theinverse of any real nonsingular n× n matrix in time O(M(n)).
90 / 102
![Page 214: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/214.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
91 / 102
![Page 215: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/215.jpg)
Least-squares Approximation
Fitting curves to given sets of data points is an important applicationof symmetric positive-definite matrices.Given
(x1, y1), (x2, y2), ..., (xm, ym)
where the yi are known to be subject to measurement errors. We wouldlike to determine a function F (x) such that:
yi = F (xi) + ηi
for i = 1, 2, ...,m
92 / 102
![Page 216: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/216.jpg)
Least-squares Approximation
ContinuationThe form of the function F depends on the problem at hand.
F (x) =n∑j=1
cjfj(x)
A common choice is fj(x) = xj−1, which means that
F (x) = c1 + c2x+ c3x2 + ...+ cnx
n−1
is a polynomial of degree n− 1 in x.
93 / 102
![Page 217: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/217.jpg)
Least-squares ApproximationContinuationLet
A =
f1(x1) f2(x1) . . . fn(x1)f1(x2) f2(x2) . . . fn(x2)
...... . . . ...
f1(xm) f2(xm) . . . fn(xm)
denote the matrix of values of the basis functions at the given points; thatis, aij = fj(xi). Let c = (ck) denote the desired size-n vector ofcoefficients. Then,
A =
f1(x1) f2(x1) . . . fn(x1)f1(x2) f2(x2) . . . fn(x2)
...... . . . ...
f1(xm) f2(xm) . . . fn(xm)
c1c2...cn
=
F (x1)F (x2)
...F (xm)
94 / 102
![Page 218: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/218.jpg)
Least-squares Approximation
ThenThus, η = Ac− y is the size of approximation errors. To minimizeapproximation errors, we choose to minimize the norm of the error vector ,which gives us a least-squares solution.
||η||2 = ||Ac− y||2 =m∑i=1
(n∑j=1
aijcj − yi
)2
ThusWe can minimize ||η|| by differentiating ||η|| with respect to each ck andthen setting the result to 0:
d||η||2dck
=m∑i=1
2(
n∑j=1
aijcj − yi
)aik = 0
95 / 102
![Page 219: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/219.jpg)
Least-squares Approximation
ThenThus, η = Ac− y is the size of approximation errors. To minimizeapproximation errors, we choose to minimize the norm of the error vector ,which gives us a least-squares solution.
||η||2 = ||Ac− y||2 =m∑i=1
(n∑j=1
aijcj − yi
)2
ThusWe can minimize ||η|| by differentiating ||η|| with respect to each ck andthen setting the result to 0:
d||η||2dck
=m∑i=1
2(
n∑j=1
aijcj − yi
)aik = 0
95 / 102
![Page 220: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/220.jpg)
Least-squares Approximation
We can put all derivativesThe n equation for k = 1, 2, ..., n
(Ac− y)TA = 0
or equivalently to
AT (Ac− y) = 0
which implies
ATAc = AT y
96 / 102
![Page 221: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/221.jpg)
Least-squares Approximation
ContinuationThe ATA is symmetric:
If A has full column rank, then ATA is positive- definite as well.Hence, (ATA)−1 exists, and the solution to equation ATAc = AT y is
c = ((ATA)−1AT )y = A+y
where the matrix A+ = ((ATA)−1AT ) is called the pseudoinverse of thematrix A.
97 / 102
![Page 222: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/222.jpg)
Least-Square ApproximationContinuationAs an example of producing a least-squares fit, suppose that we have 5data points (-1,2), (1,1),(2,1),(3,0),(5,3), shown as black dots in followingfigure
98 / 102
![Page 223: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/223.jpg)
Least-squares Approximation
ContinuationWe start with the matrix of basis-function values
A =
1 x1 x2
11 x2 x2
21 x3 x2
31 x4 x2
41 x5 x2
5
=
1 −1 11 1 11 2 41 3 91 5 25
whose pseudoinverse is
A+ =
0.500 0.300 0.200 0.100 −0.100−0.388 0.093 0.190 0.193 −0.0880.060 −0.036 −0.048 −0.036 0.060
99 / 102
![Page 224: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/224.jpg)
Matrix multiplication and matrix inversion
ContinuationMultiplying y by A+ , we obtain the coefficient vector
c =
1.200−0.7570.214
which corresponds to the quadratic polynomial
F (x) = 1.200− 0.757x+ 0.214x2
100 / 102
![Page 225: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/225.jpg)
Outline1 Introduction
Basic DefinitionsMatrix Examples
2 Matrix OperationsIntroductionMatrix MultiplicationThe InverseDeterminants
3 Improving the Complexity of the Matrix MultiplicationBack to Matrix MultiplicationStrassen’s AlgorithmThe AlgorithmHow he did it?Complexity
4 Solving Systems of Linear EquationsIntroductionLower Upper DecompositionForward and Back SubstitutionObtaining the MatricesComputing LU decompositionComputing LUP decomposition
5 ApplicationsInverting MatricesLeast-squares Approximation
6 ExercisesSome Exercises You Can Try!!!
101 / 102
![Page 226: 23 Matrix Algorithms](https://reader031.vdocuments.us/reader031/viewer/2022022412/58f2c9de1a28abf4498b45b3/html5/thumbnails/226.jpg)
Exercises
From Cormen’s book solve34.5-134.5-234.5-334.5-434.5-534.5-734.5-8
102 / 102