2
• Linear Equations– Gauss-Seidel– Jacobi
• Nonlinear Equations– Successive substitution– Newton-Raphson
3
Jacobi Iterative Method
• Goal: solve [A][x]=[b] iteratively. • Algorithm: let [A]=[A’] + [D] where
[A’]: the off-diagonal part of [A],[D]: the diagonal part of [A],then
[D][x]=[b]-[A’][x].The j-th iteration of x is computed by
[xj]=[D]-1([b]-[A’][xj-1])
4
Jacobi Iterative Method (cont.)
• Need initial guess of x.• Convergent not guaranteed.• In mathematical form
x ji
bin
m1aimx j1
m
aii, i1,2,...,nx j
i
bin
m1aimx j1
m
aii, i1,2,...,n
5
Gauss-Seidel Method
• Goal: solve [A][x]=[b] iteratively. • Algorithm: compute the j-th iteration of x
by
x ji
bii1
m1aimx j
m n
mi1aimx j1
m
aii, i1,2,...,nx j
i
bii1
m1aimx j
m n
mi1aimx j1
m
aii, i1,2,...,n
6
Gauss-Seidel Method (cont.)
• Use updated values of xij as soon as possible.
• Example: n=3
x j1
b1a12xj1
2 a13xj13
a11
x j2
b2a21xj
1a23xj1
3
a22
x j3
b3a31xj
1a32xj
2
a33
x j1
b1a12xj1
2 a13xj13
a11
x j2
b2a21xj
1a23xj1
3
a22
x j3
b3a31xj
1a32xj
2
a33
7
Example 12.1
• Solve the following by Gauss-Seidel method.
x17.850.1x20.2x3
3
x219.30.1x10.3x3
7
x371.40.3x10.2x2
10
x17.850.1x20.2x3
3
x219.30.1x10.3x3
7
x371.40.3x10.2x2
10
3x1 0.1x2 0.2x3 7.85
0.1x1 7x2 0.3x3 19.3
0.3x1 0.2x2 10x3 71.4
3x1 0.1x2 0.2x3 7.85
0.1x1 7x2 0.3x3 19.3
0.3x1 0.2x2 10x3 71.4
x1
x2
x3
32.5
7
x1
x2
x3
32.5
7Answer:
8
Comparison
• Gauss-Seidel usually but not always converges fasterthan Jacobi because the updated values of x are used as soon as possible.
9
Convergence Criteria
• Sufficient condition: diagonal dominant
• Proof:
|aii|> N
j1,ji|aij|, i1,2,...,N|aii|>
N
j1,ji|aij|, i1,2,...,N
x (n1)i
bi
aii
N
j1,ji
aij
aiix (n)
j
x (n1)i x (n)
i N
j1,ji
aij
aii(x (n)
j x (n1)j )
x (n1)i
bi
aii
N
j1,ji
aij
aiix (n)
j
x (n1)i x (n)
i N
j1,ji
aij
aii(x (n)
j x (n1)j )
10
Convergence Criteria (cont.)
• Proof (cont.):
Δx (n1)i
N
j1,ji
aij
aiiΔx (n)
j
N
j1,ji
aij
aii Δx (n)
j
Δx (n)max
N
j1,ji
aij
aii
Δx (n)max
Δx (n1)i
N
j1,ji
aij
aiiΔx (n)
j
N
j1,ji
aij
aii Δx (n)
j
Δx (n)max
N
j1,ji
aij
aii
Δx (n)max
11
Relaxation
• Purpose: to speed up converge by
• is problem specific, determined experimentally.
x newi λx new
i (1λ)x oldi , 0λ2x new
i λx newi (1λ)x old
i , 0λ2
12
Successive Substitution
f1(x1,x2,...,xn)0
f2(x1,x2,...,xn)0
fn(x1,x2,...,xn)0
x1g1(x1,x2,...,xn)
x2g2(x1,x2,...,xn)
xngn(x1,x2,...,xn)
f1(x1,x2,...,xn)0
f2(x1,x2,...,xn)0
fn(x1,x2,...,xn)0
x1g1(x1,x2,...,xn)
x2g2(x1,x2,...,xn)
xngn(x1,x2,...,xn)
x j11 g1(x
j1,x
j2,...,x
jn)
x j12 g2(x
j11 ,x j
2,...,xjn)
x j1n gn(x
j11 ,x j1
2 ,...,x jn)
x j11 g1(x
j1,x
j2,...,x
jn)
x j12 g2(x
j11 ,x j
2,...,xjn)
x j1n gn(x
j11 ,x j1
2 ,...,x jn)
13
Successive Substitution (cont.)
• Convergent Criteria (sufficient only): in an interval about the root, where initial guess is within
• May be difficult to find.
n
i1
gj
xi<1, j1,2,..n
n
i1
gj
xi<1, j1,2,..n
14
Successive Substitution (cont.)
• Example 12.2– Initial value: x1=1.5, x2=3.5 (correct answer: x1=2, x2=3)– Compare the following two cases
x 21x1x2100
x23x1x2257
x1
10x 21
x2
x2573x1x22
x 21x1x2100
x23x1x2257
x1
10x 21
x2
x2573x1x22
x 21x1x2100
x23x1x2257
x1 10x1x2
x257x2
3x1
x 21x1x2100
x23x1x2257
x1 10x1x2
x257x2
3x1
15
Newton-Raphson
• Let x1j, x2
j, …, xnj be close to the root, then
by Taylor’s expansion
f j11 f j
1n
i1(x j1
i x ji )f j
1
xi
f j12 f j
2n
i1(x j1
i x ji )f j
2
xi
f j1n f j
1n
i1(x j1
i x ji )f j
n
xi
f j11 f j
1n
i1(x j1
i x ji )f j
1
xi
f j12 f j
2n
i1(x j1
i x ji )f j
2
xi
f j1n f j
1n
i1(x j1
i x ji )f j
n
xi
17
Newton-Raphson (cont.)
0f j1
n
i1(x j1
i x ji )f j
1
xi
0f j2
n
i1(x j1
i x ji )f j
2
xi
0f j1
n
i1(x j1
i x ji )f j
n
xi
0f j1
n
i1(x j1
i x ji )f j
1
xi
0f j2
n
i1(x j1
i x ji )f j
2
xi
0f j1
n
i1(x j1
i x ji )f j
n
xi
f j1
f j2
f jn
f j1
x1
f j1
x2
f j1
xn
f j2
x1
f j2
x2
f j2
xn
f jn
x1
f jn
x2
f jn
xn
x j11 x j
1
x j12 x j
2
x j1n x j
n
f j1
f j2
f jn
f j1
x1
f j1
x2
f j1
xn
f j2
x1
f j2
x2
f j2
xn
f jn
x1
f jn
x2
f jn
xn
x j11 x j
1
x j12 x j
2
x j1n x j
n
18
Newton-Raphson (cont.)
• To sum up: [xj+1]=[xj]- [J]-1 [f j], where
[f j]
f j1
f j2
f jn
, [J]
f j1
x1
f j1
x2
f j1
xn
f j2
x1
f j2
x2
f j2
xn
f jn
x1
f jn
x2
f jn
xn
, [x j]
x j1
x j2
x jn
[f j]
f j1
f j2
f jn
, [J]
f j1
x1
f j1
x2
f j1
xn
f j2
x1
f j2
x2
f j2
xn
f jn
x1
f jn
x2
f jn
xn
, [x j]
x j1
x j2
x jn
19
Newton-Raphson (cont.)
• Shortcomings– [J] is difficult to evaluate → use finite
difference approximation.– Good initial guess is required to ensure
converge.• Alternative: nonlinear optimization
F(x)n
i1fi(x1,x2,...,xn)
2F(x)n
i1fi(x1,x2,...,xn)
2