relaxation methods for iterative solution to linear...

Relaxation Methods for Iterative Solution

to Linear Systems of Equations

Gerald Recktenwald

Portland State University

Mechanical Engineering Department

[email protected]

Primary Topics

• Basic Concepts

• Stationary Methods a.k.a. Relaxation methods

� Gauss Seidel

� Jacobi

� SOR

Relaxation Solution to Linear Systems page 1

Direct and Iterative Methods

Solve

Ax = b

where A is n × n matrix, x and b are n element column vectors.

Gaussian elimination with backward substitution is a Direct Method

• A Direct Method obtains the solution in a finite number of steps

• Solution is equivalent to

x = A−1

b

Iterative Methods obtain a sequence of approximations to the solution

xk

k = 0, 1, 2, . . .

such that (b − Axk) → 0 as k → ∞.


Residual

The residual

r = b − Ax

is zero when x is the solution to Ax = b.

For an iterative method the residual at iteration k is

rk = b − Ax

kk = 0, 1, 2, . . .

If an iterative methods converges, then rk → 0 as k → ∞.


Relaxation Methods (1)

Overview

• Simple to program

• Converges slowly for large systems of equations (large n)

• Not a useful stand alone solution method

• Key ingredient to multigrid methods

Examples

• Jacobi

• Gauss-Seidel

• SOR


Relaxation Methods (2)

Basic Idea

• Update elements of the solution vector one element at a time

• “Solve” a nodal equation by assuming other nodal values are known

N

P

S

EW

Apply the finite volume method to get

−aSφS − aWφW + aPφP − aEφE − aNφN = b

If k is the iteration counter, we “solve” for φk+1P

φk+1P

=1

aP

�b + aSφ

k

S+ aWφ

k

W+ aEφ

k

E+ aNφ

k

N

�

Values of φW , φE, φN and φS are updated by applying the

iterative formula to the cells centered at those neighbor nodes.


Matrix Notation (1)

On a structured mesh the interior nodes can

be numbered sequentially with a mapping like

i = I − 1 + (J − 2)nx

where nx is the number of cells in the x

direction and xi is the value of the ithdependent field variable which is located at

the cells with indices I and J .

I=1 I=nx+2

J=ny+2

J=12

2

3

3

i=1 i=2 i=nx

i=nx+1

x

y

i=nxny

The compass point notation

−aSφS − aWφW + aPφP − aEφE − aNφN = b

becomes

−aS,ixi−nx− aW,ixi−1 + aP,ixi − aE,ixi+1 − aW,ixi+nx

= bi


Matrix Notation (2)

The natural ordering of nodes on a two-dimensional structured mesh yields a coefficient

matrix with five diagonals. Coefficients on the diagonals correspond to aP coefficient of

φP ↔ xi and the four neighbor coefficients aS, aW , aE, and aN .

A = – aS aP – aE– aW – aN


Matrix Notation (3)

In an unstructured mesh, the coefficient matrix is also sparse, but the non-zero elements

of the coefficient matrix do not lie along diagonals.

Unstructured Mesh Non-zeros in coefficient matrix

0 10 20 30 40 50 60 70 80 90

0

10

20

30

40

50

60

70

80

90

nz = 361

To simplify the presentation, we’ll use a structured mesh.


Relaxation Methods

Use relaxation methods to solve

Ax = b

Relaxation methods do not work for all A, but they do work for standard finite-difference

and finite-volume discretizations of the Poisson equation.

• Jacobi

• Gauss-Seidel

• SOR – Successive Over-relaxation


Jacobi Iteration (1)

A discrete model of the Poisson equation yields a system of equations that can be written

Ax = b. The equation for the ith nodal value is

n�

j=1

aijxj = bi

where n is the total number of nodes in the domain.

Extract the diagonal term from the sum

aiixi +n�

j=1j �=i

aijxj = bi

The preceding equation will not be satisfied until, in the limit as the number of iterations

increased to ∞.



“Solve” for the next guess xi.

xi =1

aii

�b −

n�

j=1j �=i

aijxj

�(�)

In terms of the compass-point notation for the two-dimensional finite-volume method, the

Equation (�) is

φ(k+1)P

=1

aP

�b + aSφ

(k)S

+ aWφ(k)W

+ aEφ(k)E

+ aNφ(k)N

�

Equation (�) is not really the solution for xi until all the other xj are known. In other

words, the value of xi from Equation (�) will be correct only when the system ofequations Ax = b is solved.

If matrix A has favorable properties, the value of xi from Equation (�) will be closer to

the solution than the previous guess at xi.



Let k be the iteration counter. The Jacobi iteration formula is

x(k+1)i

=1

aii

�bi −

n�

j=1j �=i

aijx(k)j

�i = 1, . . . , n

This formula implies that two copies of the x vector are maintained in memory: one for

iteration k and one for iteration k + 1.

for k=1:maxit

% -- For nodes away from the boundaries

for i=(nx+1):n-nx

xnew(i) = (b(i) + as(i)*x(i-nx) + aw(i)*x(i-1) ...

+ an(i)*x(i+nx) + ae(i)*x(i+1) ) / ap(i);

end

x = xnew;

end



The Jacobi iterative formula can be written as a matrix equation.

Let D = diag(a11, . . . , ann) and B = D − A then

x(k+1)i

=1

aii

�b −

n�

j=1j �=i

aijx(k)j

�

is equivalent to

x(k+1) = D−1

�b + Bx(k)

�(1)

where x(k+1)and x(k)

are the solution vectors at iteration k + 1 and k, respectively.



The matrix form of the iteration can be rewritten

x(k+1) = Hx(k) + d (2)

where

H = D−1

B

d = D−1b

The matrix formulation is primarily useful in the analysis of iterative methods and in toy

implementations in Matlab. The eigenvalues of H determine the convergence rate of

the iterations.


Stopping Criteria (1)

Recall that the goal is to solve

Ax = b

where A is n × n matrix, x and b are n element column vectors.

Iterative Methods obtain a sequence of approximations to the solution

xk

k = 0, 1, 2, . . .

such that b − Axk → 0 as k → ∞.

The residual

r = b − Ax

is zero when x is the solution to Ax = b.


Stopping Criteria (2)

For an iterative method the residual at iteration k is

rk = b − Ax

kk = 0, 1, 2, . . .

If an iterative methods converges, then rk → 0 as k → ∞.

Since r is a vector, what does it mean that rk → 0?

Use a convenient vector norm to test whether �r� is small enough.



Solution to a toy problem

∂2T

∂x2+

∂2T

∂y2= 0

on 0 ≤ x ≤ L, 0 ≤ y ≤ W

T = 0 on three boundaries

uniform q on fourth boundary

Solution

00.5

11.5

00.5

11.5

23

3.5

4

4.5

Solution is obtained by Matlab code demoJacobi



Convergence of Jacobi iterations slow as mesh is refined.

0 200 400 600 800 1000 1200 140010−4

10−3

10−2

10−1

100

Iteration

|| r |

| 1 / ||

r 0 ||1

8 x 816 x 1632 x 32


Gauss-Seidel Iteration (1)

In the Jacobi method x(k+1)

is obtained from the frozen values of x(k)

.

The Gauss-Seidel method uses a similar formula for updating x(k+1)i

, but the new value of

x(k+1)i

is used as soon as it is available

x

y

values from

iteration k

values from

iteration k+1



Replace nodal values as sweep progresses through the domain

for k=1:maxit

% -- For nodes away from the boundaries

for i=(nx+1):n-nx

x(i) = (b(i) + as(i)*x(i-nx) + aw(i)*x(i-1) ...

+ an(i)*x(i+nx) + ae(i)*x(i+1) ) / ap(i);

end

end



Consider the sweep through the nodal values in order of increasing i:

x(k+1)1 =

1

a11

�b1 −

n�

j=2

a1jx(k)j

�

x(k+1)2 =

1

a22

�b2 − a21x

(k+1)1 −

n�

j=3

a1jx(k)j

�

x(k+1)3 =

1

a33

�b3 − a31x

(k+1)1 − a32x

(k+1)2 −

n�

j=4

a1jx(k)j

�

...

x(k+1)i

=1

aii

�bi −

i−1�

j=1

aijx(k+1)i

−n�

j=i+1

a1jx(k)j

�(3)



Coefficients thatmultiply new values

Coefficients thatmultiply old values

i

A x = b

new

val

ues

old

val

ues

=

Again, let

D = diag(a11, . . . , ann)

and define

L = lower triangular part of A

U = upper triangular part of A

Then Gauss-Seidel iterations correspond

to the splitting of A

A = D − L − U

Note that A �= LU , i.e.,, L and U are

not the usual factors of A.



Equation (3) can be rewritten as

aiix(k+1)i

= bi −i−1�

j=1

aijx(k+1)i

−n�

j=i+1

a1jx(k)j

or

aiix(k+1)i

−i−1�

j=1

aijx(k+1)i

= bi −n�

j=i+1

a1jx(k)j

or

Dx(k+1) − Lx

(k+1) = b + Ux(k)

=⇒ (D − L)x(k+1) = Ux(k) + b

or

x(k+1) = (D − L)−1(Ux

(k) + b) (4)



Note: The order in which the nodes are processed in the Gauss-Seidel iterations will

affect the path toward the solution. In other words, for two different orderings of the

nodes, the intermediate iterates of x wil be different.

The Jacobi iteration does not depend on the order in which the nodes are numbered.

Much of the classical theory of relaxation methods assumes that the nodes and numbered

in natural order.


SOR: Successive Over-Relaxation (1)

Let x̂(k+1)i

be the updated value of xi from the Gauss-Seidel formula, viz.

x̂(k+1)i

=1

aii

�bi −

�

j<i

aijx(k+1)j

−�

j>i

aijx(k)j

�(5)

Instead of taking x̂(k+1)i

as the value of xi at the k + 1 step, use

x(k+1)i

= ωx̂(k+1)i

+ (1 − ω)x(k)i

(6)

where ω is a scalar weighting factor. The basic step in Equation (5) is called relaxation.

ω = 1 for Gauss-Seidel

ω < 1 causes the iterations to more slowly move toward the solution.

This is called under-relaxation.

ω > 1 causes the iterations to accelerate, i.e. to be more aggressive.

This is called over-relaxation.


Relaxation as an update

We can rewrite Equation (6) as

x(k+1)i

= x(k)i

+ ω

�x̂(k+1)i

− x(k)i

�(7)

or

x(k+1)i

= x(k)i

+ ω∆x(k+1)i

(8)

where

∆x(k+1)i

= x̂(k+1)i

− x(k)i

∆x(k+1)i

is the update to xi in iteration k.

∆x(k+1)i

is the direction in which the solution is changing.

ω∆x(k+1)i

is a magnified (ω > 1) or reduced (ω < 1) change to the solution.



Substitute Equation (6) into Equation (5)

x(k+1)i

= (1 − ω)xk

i+

ω

aii

�bi −

�

j<i

aijx(k+1)j

−�

j>i

aijx(k)j

�(9)

Multiply through by aii and rearrange to get

aiix(k+1)i

+ ω

�

j<i

aijx(k+1)j

= (1 − ω)aiixk

i− ω

�

j>i

aijx(k)j

+ ωbi

Use the splitting A = D − L − U and the preceding equation becomes

Dx(k+1) − ωLx(k+1) = (1 − ω)Dx(k) + ωUx(k) + ωb

(D − ωL)x(k+1) =�(1 − ω)Dx(k) + ωU

�x(k) + ωb

x(k+1) = (D − ωL)−1�(1 − ω)Dx(k) + ωU�x(k) + ω(D − ωL)−1b



The update equation is

x(k+1) = (D − ωL)−1�(1 − ω)Dx(k) + ωU�x(k) + ω(D − ωL)−1b

Thus

x(k+1) = Hx(k) + d

where

H = (D − ωL)−1�(1 − ω)Dx(k) + ωU�

d = ω(D − ωL)−1b

The speed of convergence depends on the eigenvalues of H.


Compare Relaxation Convergence

On a coarse 8 × 8 mesh

0 20 40 60 80 100 120100

101

102

103

104

Iteration

||r|| 1

JacobiGauss SeidelSOR


Compare Relaxation Convergence

On a coarse 16 × 16 mesh

0 50 100 150 200 250 300 350 400101

102

103

104

105

Iteration

||r|| 1

JacobiGauss SeidelSOR


relaxation methods for iterative solution to linear...

Documents