lecture topic: systems of linear equations - ibm › researcher › files › ie... ·...

Lecture Topic: Systems of Linear Equations

Introduction: systems of linear equationsWe examine both iterative and direct methods for solving equations

Ax = b (1)

where x , b ∈ Rn, A ∈ MnR is an n × n matrix,x is an unknown vector (to be found) and b and A are known.

Solving systems of linear equations is still the most important problem incomputational mathematics, because it is used as a sub-problem in solving otherproblems.

Algorithms that solve non-linear systems commonly use linear approximations,which give rise to systems of linear equations.

Algorithms that optimise over feasible sets given by linear and non-linear equalitiesand inequalities commonly solve systems related to first-order optimalityconditions iteratively, which give rise to systems of linear equations.

Jakub Marecek and Sean McGarraghy (UCD) Numerical Analysis and Software October 8, 2015 2 / 1

Ax = b (1)

Mathematics and Algorithms

In Quantitative Methods you have learned the mathematical ideas behind directand iterative approaches to solving (1). You should understand:

Theorem

The following are equivalent for any n × n matrix A:

Ax = b has a unique solution of all b ∈ R.

Ax = 0 implies x = 0.

A−1 exists.

det(A) 6= 0.

rank(A) = n.

The full rank of A is also our assumption throughout the chapter.

Here, we build on these and analyse the related algorithms, focussing first onconditioning, second on the stability of direct methods, and third on theconvergence and stability of iterative methods.

This still leaves much unexplained, including conjugate gradients (CG),generalised minimal residuals (GMRES), and preconditioning, i.e. methods forchanging the condition.

Here, we build on these and analyse the related algorithms, focussing first onconditioning, second on the stability of direct methods, and third on theconvergence and stability of iterative methods.

This still leaves much unexplained, including conjugate gradients (CG),generalised minimal residuals (GMRES), and preconditioning, i.e. methods forchanging the condition.

Condition of a System of Linear Equations

Condition suggests how changes in A and b, the “instance” of the problem, affectthe solution x , using any algorithm.

We will examine errors in A and b separately.

It turns out that in both cases the condition number of the matrix A plays a role.

An ExampleConsider the system of linear equations

x1 + 0.99x2 = 1.99

0.99x1 + 0.98x2 = 1.97.

The true solution is x1 = 1 and x2 = 1 but x1 = 3.0000 and x2 = −1.0203 gives

x1 + 0.99x2 = 1.989903

0.99x1 + 0.98x2 = 1.970106.

Thus, a small change in the problem data, a change in the vector b from(1.991.97

(1.9899031.970106

leads to a large change in the solution:this is our criterion for ill-conditioning.

x1 + 0.99x2 = 1.99

0.99x1 + 0.98x2 = 1.97.

x1 + 0.99x2 = 1.989903

0.99x1 + 0.98x2 = 1.970106.

(1.9899031.970106

x1 + 0.99x2 = 1.99

0.99x1 + 0.98x2 = 1.97.

x1 + 0.99x2 = 1.989903

0.99x1 + 0.98x2 = 1.970106.

(1.9899031.970106

x1 + 0.99x2 = 1.99

0.99x1 + 0.98x2 = 1.97.

x1 + 0.99x2 = 1.989903

0.99x1 + 0.98x2 = 1.970106.

(1.9899031.970106

An Example

−1 0 1 2 3−1

An Example

0.9999 1.0000 1.00010.9999

1.0000

1.0001

Perturbation of bLet the right-hand-side b be perturbed by δb. So we want to find the solution of

A(x + δx) = b + δb. (2.1)

‖ ‖ denotes a vector or matrix norm, according to context. By (2.1), Aδx = δb, so

δx = A−1δb ⇒ ‖δx‖ ≤ ‖A−1‖ ‖δb‖ (a sharp bound). (2.2)

Since b = Ax , the properties of matrix norms again give

‖b‖ ≤ ‖A‖ ‖x‖. (2.3)

Hence, combining (2.2) and (2.3): each LHS≤RHS, so∏

LHS≤∏RHS:

‖δx‖ ‖b‖ ≤ ‖A‖ ‖A−1‖ ‖x‖ ‖δb‖and assuming b 6= 0 we get

‖δx‖‖x‖ ≤ ‖A‖ ‖A

−1‖‖δb‖‖b‖i.e., (rel. error in x) ≤ ‖A‖ ‖A−1‖(rel. error in b).

A(x + δx) = b + δb. (2.1)

‖b‖ ≤ ‖A‖ ‖x‖. (2.3)

LHS≤∏RHS:

‖δx‖‖x‖ ≤ ‖A‖ ‖A

A(x + δx) = b + δb. (2.1)

‖b‖ ≤ ‖A‖ ‖x‖. (2.3)

LHS≤∏RHS:

‖δx‖‖x‖ ≤ ‖A‖ ‖A

A(x + δx) = b + δb. (2.1)

‖b‖ ≤ ‖A‖ ‖x‖. (2.3)

LHS≤∏RHS:

‖δx‖‖x‖ ≤ ‖A‖ ‖A

A(x + δx) = b + δb. (2.1)

‖b‖ ≤ ‖A‖ ‖x‖. (2.3)

LHS≤∏RHS:

‖δx‖‖x‖ ≤ ‖A‖ ‖A

A(x + δx) = b + δb. (2.1)

‖b‖ ≤ ‖A‖ ‖x‖. (2.3)

LHS≤∏RHS:

‖δx‖‖x‖ ≤ ‖A‖ ‖A

A(x + δx) = b + δb. (2.1)

‖b‖ ≤ ‖A‖ ‖x‖. (2.3)

LHS≤∏RHS:

‖δx‖‖x‖ ≤ ‖A‖ ‖A

The Condition Number of a MatrixThus, the quantity ‖A‖ ‖A−1‖ measures the relative change in solution for a givenrelative change in problem: it measures the relative condition of the system oflinear equations problem.

Definition

Given a matrix norm ‖ ‖, the condition number of matrix A is

condrel(A) = ‖A‖‖A−1‖.

This depends on the norm used; but, since the underlying vector norms only differby a fixed multiplicative constant for a given n (all norms on Rn are equivalent),all measures of condition number are equally good.

We can interpret condrel(A) as:

the amount a relative error in b is magnified in the solution vector x ; or

the distortion A produces when applied to the unit sphere; or

how “close” A (and indeed A−1) is to being a singular matrix.

Definition

The Spectral Condition Number of a MatrixDefinition

We also define the spectral condition number of A as

cond*rel(A) :=

maxλ∈σ(A) |λ|minλ∈σ(A) |λ|

Here σ(A), the spectrum of A, is the set of all eigenvalues of A.

If λ is an eigenvalue of A, its modulus or length |λ| is the factor by which aλ-eigenvector is expanded (if |λ| > 1) or contracted (if |λ| < 1). Thus

ρ(A) = maxλ∈σ(A) |λ|, the spectral radius, is the largest factor by which Amultiplies an eigenvector, whileminλ∈σ(A) |λ| is the smallest factor by which A multiplies an eigenvector.

The ratio cond*rel(A) is thus a measure of the distortion produced by A:

how great is the difference in expansion/contraction of eigenvectors that A cancause.Jakub Marecek and Sean McGarraghy (UCD) Numerical Analysis and Software October 8, 2015 11 / 1

cond*rel(A) :=

An Example

In the example above, the related matrix

(1.00 .99.99 .98

has eigenvalues λ1 = 1.98, λ2 = −0.00005 of the characteristic equation,

det(A− λI ) = det

(1.00− λ .99.99 .98− λ

)= (1− λ)(.98− λ)− .992

= λ2 − 1.98λ+ .98− .9801 = λ2 − 1.98λ− .0001.

Thus the spectral condition number is cond*rel(A) = |1.98|/| − 0.00005| =39,600.

Hence, this matrix is very ill-conditioned.

An Example

In the example above, the related matrix

(1.00 .99.99 .98

has eigenvalues λ1 = 1.98, λ2 = −0.00005 of the characteristic equation,

det(A− λI ) = det

(1.00− λ .99.99 .98− λ

)= (1− λ)(.98− λ)− .992

= λ2 − 1.98λ+ .98− .9801 = λ2 − 1.98λ− .0001.

Thus the spectral condition number is cond*rel(A) = |1.98|/| − 0.00005| =39,600.

Hence, this matrix is very ill-conditioned.

Properties of the Condition NumberThe condition number condrel(A) is bounded below by 1:this is seen by noting that ‖I‖ = 1 for any norm and

1 = ‖I‖ = ‖AA−1‖ ≤ ‖A‖‖A−1‖ = condrel(A).

Each norm-based condition number is also bounded below by the spectralcondition number of A:

1 ≤ cond*rel(A) ≤ condrel(A)

for any norm.

Thus the spectral condition number is the smallest measure of relative conditionof the system of linear equations problem.

1 = ‖I‖ = ‖AA−1‖ ≤ ‖A‖‖A−1‖ = condrel(A).

for any norm.

1 = ‖I‖ = ‖AA−1‖ ≤ ‖A‖‖A−1‖ = condrel(A).

for any norm.

Perturbation of AIf A is perturbed by δA then we have

b = (A + δA)(x + δx)

= Ax + Aδx + δAx + δAδx

⇒ Aδx = −δA(x + δx)

⇒ δx = −A−1δA(x + δx)

Taking norms and using the triangle inequality we have∥∥δx∥∥ =∥∥A−1δA(x + δx)

∥∥≤ ‖A−1‖‖δA‖(‖x‖+ ‖δx‖)

⇒ ‖δx‖(1− ‖A−1‖ ‖δA‖

)≤ ‖A−1‖ ‖δA‖ ‖x‖.

Thus‖δx‖‖x‖ ≤ ‖A−1‖ ‖δA‖

1− ‖A−1‖ ‖δA‖ =‖A‖ ‖A−1‖

(1− ‖A−1‖ ‖δA‖)‖δA‖‖A‖

b = (A + δA)(x + δx)

⇒ δx = −A−1δA(x + δx)

∥∥≤ ‖A−1‖‖δA‖(‖x‖+ ‖δx‖)

⇒ ‖δx‖(1− ‖A−1‖ ‖δA‖

)≤ ‖A−1‖ ‖δA‖ ‖x‖.

1− ‖A−1‖ ‖δA‖ =‖A‖ ‖A−1‖

(1− ‖A−1‖ ‖δA‖)‖δA‖‖A‖

b = (A + δA)(x + δx)

⇒ δx = −A−1δA(x + δx)

∥∥≤ ‖A−1‖‖δA‖(‖x‖+ ‖δx‖)

⇒ ‖δx‖(1− ‖A−1‖ ‖δA‖

)≤ ‖A−1‖ ‖δA‖ ‖x‖.

1− ‖A−1‖ ‖δA‖ =‖A‖ ‖A−1‖

(1− ‖A−1‖ ‖δA‖)‖δA‖‖A‖

The Condition Number Again. . .

Since 1 = ‖I‖ = ‖AA−1‖ ≤ ‖A‖‖A−1‖ we have 1‖A‖ ≤ ‖A−1‖.

Thus, if ‖A−1‖ ‖δA‖ � 1 (and so ‖δA‖/‖A‖ � 1),

then 1− ‖A−1‖ ‖δA‖ ≈ 1 and we have

‖δx‖‖x‖ ≤ condrel(A)

‖δA‖‖A‖ .

Thus, for a small perturbation of A, we again have that condition numbermeasures the relative condition of the system of linear equations problem.

A similar result can be derived for the case where both A and b are perturbed.

‖δA‖‖A‖ .

“How Close to Singular?”

Theorem

If A is non-singular, and‖δA‖‖A‖ <

condrel(A)

then A + δA is also non-singular.

This theorem tells us that the condition number measures the distance from A tothe nearest singular matrix:it is a better measure than the determinant of“how close to singularity” a matrix is.

Theorem

condrel(A)

Theorem

condrel(A)

Theorem

condrel(A)

Errors and Residuals

There are two common ways to measure the discrepancy between the truesolution x and the computed solution x :

Error δx = x − x

Residual r = b − Ax

If A is invertible, and either δx or r is zero, then both must be zero.In many applications, we want to solve Ax = b so that r , the difference betweenthe LHS and RHS, is small, i.e., so that ‖r‖ = ‖b − Ax‖ is small.

Intuitively, we can think of the residual as follows:if you have a computed solution x to a system of linear equations and you knowthe exact solution x , then you know the error δx = x − x ;but if you don’t know the solution x beforehand then the residual is a measurealong a different axis of how close you are, r = b − Ax .

Error δx = x − x

Cond(A), r and Errors in xLet x be the computed solution to Ax = b.Then δx = x − x and r = b − Ax , giving

Aδx = Ax − Ax = b − Ax = r

⇒ δx = A−1r .

Thus ‖δx‖ ≤ ‖A−1‖ ‖r‖ (property of matrix norms) [1].

Similarly ‖b‖ ≤ ‖A‖ ‖x‖

‖x‖ ≤‖A‖‖b‖ [2].

It follows that‖δx‖‖x‖ ≤ ‖A‖ ‖A

−1‖ ‖r‖‖b‖ (combining [1] and [2]).

Thus‖δx‖‖x‖ ≤ condrel(A)

‖r‖‖b‖

The Effect of Ill-conditioned A

The conclusion is:if A is ill-conditioned then small ‖r‖ does not imply small ‖δx‖/‖x‖.(We’ll see there is a similar conclusion for solutions of non-linear equations:if the problem is ill-conditioned,then a small “residual” |f (xk)| does not mean that |xk − xk−1| is small.)

Solutions with small residuals,these can lead to large errors in x if A is ill-conditioned.

Special SystemsLet us consider the following types of matrices:

Symmetric: A = AT

Positive definite: xTAx > 0 for all x 6= 0 and therefore xTAx = λxT x > 0and eigenvalues λ > 0.Diagonally dominant (DD): The element on the diagonal is larger or equal tothe sum of the other elements in the row, i.e., |aii | ≥

∑j 6=i |aij |.

Strictly DD: The same, except for the strict inequality, i.e., |aii | >∑

j 6=i |aij |.Upper triangular: for aii 6= 0

a11 a12 · · · · · · a1n

0 a22...

... 0. . .

......

.... . .

...0 0 ann

Recall that if a matrix is in echelon form (e.g., upper triangular)the first non-zero entry in a row is called the pivot for that row:here akk is the pivot for the kth row.

Special SystemsLet us consider the following types of matrices:

Symmetric: A = AT

Positive definite: xTAx > 0 for all x 6= 0 and therefore xTAx = λxT x > 0and eigenvalues λ > 0.Diagonally dominant (DD): The element on the diagonal is larger or equal tothe sum of the other elements in the row, i.e., |aii | ≥

∑j 6=i |aij |.

Strictly DD: The same, except for the strict inequality, i.e., |aii | >∑

j 6=i |aij |.Upper triangular: for aii 6= 0

a11 a12 · · · · · · a1n

0 a22...

... 0. . .

......

.... . .

...0 0 ann

Recall that if a matrix is in echelon form (e.g., upper triangular)the first non-zero entry in a row is called the pivot for that row:here akk is the pivot for the kth row.

An Overview of the Algorithms

Direct methods for solving Ax = b apply elementary matrix operations to A andb, giving a transformed problem A′x ′ = b′ which is easily solved for x ′. Withindirect methods:

In Gauss-Jordan, multiples of a pivot row are subtracted from other rows,such that one obtains an upper triangular matrix first, and an identity matrixnext. Gauss-Jordan works (with appropriate pivot) on any matrix, but isstable only for diagonally dominant or positive-definite matrices.

Gauss-Jordan is also closely related to the LU and LUP decomposition, whereU stands for upper triangular matrix and L stands for upper triangular matrix.

On symmetric positive definite matices, one can also use other decompositionmethods (e.g., Cholesky, QR), which are stable and faster.

Iterative methods successively improve an initial guess until it becomessatisfactory.

Iterative methods for systems of linear equations are best understood as means ofsolving an associated optimisation problem.

Let us have a quadric f := 12x

TAx + bT x + c with A positive definite. Wheneverthe first-order optimality conditions of minx∈Rn f (x) are satisfied, we have Ax = b.

Within iterative methods:

Jacobi method is guaranteed to converge if A is strictly diagonally dominant.

Gauss-Seidel is guaranteed to converge if A is either diagonally dominant orsymmetric positive semidefinite.

Many other algorithms (CG, GMRES) work on symmetric positive-definitematrices.

In a number of applications, iterative methods are preferred to direct methods,especially when the coefficient matrix A is sparse or structured.

Gauss-JordanRecall that this method uses a sequence of elementary matrix operations totransform the square system Ax = b into an upper triangular system Ux = b′,which is then solved using back substitution.

We use a superscript in parentheses to denote the stage: x(k)i denotes the value

for xi at the kth stage and A(k) denotes the matrix A at this stage.

At stage k we have:

a(1)11 a

(1)12 · · · a

(1)1k · · · a

(1)1n b

0 a(2)22 · · · a

(2)2k · · · a

(2)2n b

.... . .

0 · · · · · · a(k)kk · · · a

(k)kn b

......

0 · · · · · · a(k)nk · · · a

(k)nn b

=(A(k) b(k)

Gauss-JordanRecall that this method uses a sequence of elementary matrix operations totransform the square system Ax = b into an upper triangular system Ux = b′,which is then solved using back substitution.

We use a superscript in parentheses to denote the stage: x(k)i denotes the value

for xi at the kth stage and A(k) denotes the matrix A at this stage.

At stage k we have:

a(1)11 a

(1)12 · · · a

(1)1k · · · a

(1)1n b

0 a(2)22 · · · a

(2)2k · · · a

(2)2n b

.... . .

0 · · · · · · a(k)kk · · · a

(k)kn b

......

0 · · · · · · a(k)nk · · · a

(k)nn b

=(A(k) b(k)

What Gauss-Jordan does at Stage kThe elements a

(k)k+1,k , a

(k)k+2,k ,. . . , a

(k)nk are eliminated by subtracting the following

multiples of row k from rows k + 1, k + 2, . . . , n:

mk+1,k :=a

(k)k+1,k

a(k)kk

, mk+2,k :=a

(k)k+2,k

a(k)kk

, . . . , mn,k :=a

(k)n,k

a(k)kk

We have in general, assuming that a(k)kk 6= 0, the (i , k) multiplier

mik :=a

a(k)kk

i = k + 1, . . . , n

and, for all i , j = k + 1, . . . , n,

a(k+1)ij = a

(k)ij −mika

(k)kj ,

b(k+1)i = b

(k)i −mikb

(k)k .

(k)k+1,k , a

(k)k+2,k ,. . . , a

mk+1,k :=a

(k)k+1,k

a(k)kk

, mk+2,k :=a

(k)k+2,k

a(k)kk

, . . . , mn,k :=a

(k)n,k

a(k)kk

mik :=a

a(k)kk

i = k + 1, . . . , n

and, for all i , j = k + 1, . . . , n,

a(k+1)ij = a

(k)ij −mika

(k)kj ,

b(k+1)i = b

(k)i −mikb

(k)k .

(k)k+1,k , a

(k)k+2,k ,. . . , a

mk+1,k :=a

(k)k+1,k

a(k)kk

, mk+2,k :=a

(k)k+2,k

a(k)kk

, . . . , mn,k :=a

(k)n,k

a(k)kk

mik :=a

a(k)kk

i = k + 1, . . . , n

and, for all i , j = k + 1, . . . , n,

a(k+1)ij = a

(k)ij −mika

(k)kj ,

b(k+1)i = b

(k)i −mikb

(k)k .

(k)k+1,k , a

(k)k+2,k ,. . . , a

mk+1,k :=a

(k)k+1,k

a(k)kk

, mk+2,k :=a

(k)k+2,k

a(k)kk

, . . . , mn,k :=a

(k)n,k

a(k)kk

mik :=a

a(k)kk

i = k + 1, . . . , n

and, for all i , j = k + 1, . . . , n,

a(k+1)ij = a

(k)ij −mika

(k)kj ,

b(k+1)i = b

(k)i −mikb

(k)k .

(k)k+1,k , a

(k)k+2,k ,. . . , a

mk+1,k :=a

(k)k+1,k

a(k)kk

, mk+2,k :=a

(k)k+2,k

a(k)kk

, . . . , mn,k :=a

(k)n,k

a(k)kk

mik :=a

a(k)kk

i = k + 1, . . . , n

and, for all i , j = k + 1, . . . , n,

a(k+1)ij = a

(k)ij −mika

(k)kj ,

b(k+1)i = b

(k)i −mikb

(k)k .

(k)k+1,k , a

(k)k+2,k ,. . . , a

mk+1,k :=a

(k)k+1,k

a(k)kk

, mk+2,k :=a

(k)k+2,k

a(k)kk

, . . . , mn,k :=a

(k)n,k

a(k)kk

mik :=a

a(k)kk

i = k + 1, . . . , n

and, for all i , j = k + 1, . . . , n,

a(k+1)ij = a

(k)ij −mika

(k)kj ,

b(k+1)i = b

(k)i −mikb

(k)k .

(k)k+1,k , a

(k)k+2,k ,. . . , a

mk+1,k :=a

(k)k+1,k

a(k)kk

, mk+2,k :=a

(k)k+2,k

a(k)kk

, . . . , mn,k :=a

(k)n,k

a(k)kk

mik :=a

a(k)kk

i = k + 1, . . . , n

and, for all i , j = k + 1, . . . , n,

a(k+1)ij = a

(k)ij −mika

(k)kj ,

b(k+1)i = b

(k)i −mikb

(k)k .

A Picture of the Matrix at Stage kNote that rows 1, . . . , k will not change from stage k + 1 onwards.

Reduce to zero

part of matrix that changes

Figure : Gauss-Jordan: changes at stage k.

Gauss-Jordan1 def GaussJordan(A, b, pivoting = noPivot):

(rows, cols) = A.shapefor row in range(0, rows-1):pivot = pivoting(A, row)if abs(A[pivot, row]) < 1e-8: raise ValueError()

6 if pivot != row:A[[row, pivot],:] = A[[pivot, row],:]b[[row, pivot]] = b[[pivot, row]]

for i in range(row+1, rows):if abs(A[row, row]) < 1e-8: raise ValueError()

11 factor = A[i, row] / A[row, row]A[i, row+1:rows] = A[i, row+1:rows] -

factor*A[row, row+1:rows]b[i] = b[i] - factor*b[row]

Gauss-Jordan

The back-substitution can be written as:

for k in range(rows-1,-1,-1):2 b[k] = (b[k] - dot(A[k, k+1:rows], b[k+1:rows])) /

A[k, k]return b

Analysis of Gauss-JordanSee that Line 12 performs O(n) “multiply–accumulate” operations for n rows andn. If we see “multiplyaccumulate” as 1 operation, the number S(n) of operationsperformed is:

S(n) =n−1∑k=1

n∑i=k+1

n∑j=k+1

=n−1∑k=1

n∑i=k+1

(n − k)

=n−1∑k=1

(n − k)2

= (n − 1)2 + (n − 2)2 + · · ·+ 22 + 12

= n(n − 1)(2n − 1)/6 ≈ n3/3 for large n.

Hence Gauss-Jordan is a Θ(n3) process. (∑n−1

k=1 k2 = 1

6n(n − 1)(2n − 1) byinduction.)

Analysis of Gauss-JordanSee that Line 12 performs O(n) “multiply–accumulate” operations for n rows andn. If we see “multiplyaccumulate” as 1 operation, the number S(n) of operationsperformed is:

S(n) =n−1∑k=1

n∑i=k+1

n∑j=k+1

=n−1∑k=1

n∑i=k+1

(n − k)

=n−1∑k=1

(n − k)2

= (n − 1)2 + (n − 2)2 + · · ·+ 22 + 12

= n(n − 1)(2n − 1)/6 ≈ n3/3 for large n.

Hence Gauss-Jordan is a Θ(n3) process. (∑n−1

k=1 k2 = 1

6n(n − 1)(2n − 1) byinduction.)

A Perspective on Gauss-Jordan

To put Θ(n3) into perspective, consider a single computer, which can sustain theperformance of 1011 operations per second (“100 gigaFLOPS”).

For a 10000× 10000 matrix, you need 1012 operations, or 10 seconds.

For a 100000× 100000 matrix, you need 1015 operations, or under 3 hours, if youcan store the 80 GB in RAM.

For a 1000000× 1000000 matrix, you need 1018 operations, or over 115 days, ifyou can store the 8 TB in RAM.

As you can test using your own laptopn, this is a very optimistic estimate.

The Net Effect. . .

Gauss-Jordan transforms the original system Ax = b to upper triangular form:

(1)11 a

(1)12 · · · a

0 a(2)22

......

. . ....

0 0 · · · a(n)nn

x2...xn

b(2)2...

This system of equations can now be solved using back substitution.

The Net Effect. . .

Gauss-Jordan transforms the original system Ax = b to upper triangular form:

(1)11 a

(1)12 · · · a

0 a(2)22

......

. . ....

0 0 · · · a(n)nn

x2...xn

b(2)2...

This system of equations can now be solved using back substitution.

Observations on Gauss-Jordan

Assumes a(k)kk 6= 0: but in fact since A is invertible, we could always swap row

k with a later row to get a(k)kk 6= 0 (see later).

A and b are overwritten.

The 0’s beneath the pivot element are not calculated.They are ignored, as they are known to be zero.

Thus the storage space for these zeros could be used for something else. . .

An extra matrix is not needed to store the mik ’s.They can be stored in place of the zeros.

The operations on b can be done separately, once we have stored the mik ’s.

Because of the last observation we may now solve for any b without goingthrough the elimination calculations again.

Gauss-Jordan with Varying b′

We solved Ax = b using Gauss-Jordan which required elementary row operationsto be performed on both A and b.

If we are required to solve the equation Ax = b′ then we would need to performexactly the same operations because these are determined by the elements of Aonly, and A is the same in both equations.

Hence if we have stored the multipliers mik we need to perform only thelast-but-one line of Gauss-Jordan, i.e.,

bi := bi −mikbk , k = 1, . . . , n − 1, i = k + 1, . . . , n.

The LU Decomposition of A

If at each stage k of Gauss-Jordan we store mik in those cells of A that becomezero then the A matrix after elimination would be as follows

a(1)11 a

(1)12 · · · a

m21 a(2)22

......

. . ....

mn1 mn2 · · · a(n)nn

LU Decomposition of AWe define the upper and unit lower triangular parts as

U = (uij) =

(1)11 a

(1)12 · · · a

0 a(2)22

......

. . ....

0 0 · · · a(n)nn

, L = (`ij) =

1 0 · · · 0

m21 1...

.... . .

...mn1 mn2 · · · 1

That is, for all i , j ∈ {1, . . . , n},

(i)ij if i ≤ j

0 otherwise

mij if i > j1 if i = j0 otherwise

U = (uij) =

(1)11 a

(1)12 · · · a

0 a(2)22

......

. . ....

0 0 · · · a(n)nn

, L = (`ij) =

1 0 · · · 0

m21 1...

.... . .

...mn1 mn2 · · · 1

(i)ij if i ≤ j

0 otherwise

U = (uij) =

(1)11 a

(1)12 · · · a

0 a(2)22

......

. . ....

0 0 · · · a(n)nn

, L = (`ij) =

1 0 · · · 0

m21 1...

.... . .

...mn1 mn2 · · · 1

(i)ij if i ≤ j

0 otherwise

U = (uij) =

(1)11 a

(1)12 · · · a

0 a(2)22

......

. . ....

0 0 · · · a(n)nn

, L = (`ij) =

1 0 · · · 0

m21 1...

.... . .

...mn1 mn2 · · · 1

(i)ij if i ≤ j

0 otherwise

An Unexpected Fact: A = LUTheorem (LU Decomposition)

If L = (`ij) and U = (uij) are the upper and lower triangular matrices generated

by Gauss-Jordan, assuming a(k)kk 6= 0 at each stage, then

A = (aij) = LU, that is, aij =n∑

`ikukj

whereukj = a

(k)kj , k ≤ j , in particular, ukk = a