cise301_topic3kfupm1 se301: numerical methods topic 3: solution of systems of linear equations...

CISE301_Topic3 KFUPM 1

SE301: Numerical Methods

Topic 3: Solution of Systems of Linear

Equations Lectures 12-17:

KFUPM

Read Chapter 9 of the textbook


Lecture 12Vector, Matrices, and

Linear Equations


VECTORS

1

0

0

0

,

0

1

0

0

,

0

0

1

0

,

0

0

0

1

1

2tor column vec 241 vector row

:Examples

numbers ofarray ldimensiona one a:Vector

4321 eeeevectorsIdentity


MATRICES

1200

1410

0143

0021

lTridiagona,

6000

0000

0040

0001

diagonal

10

01matrix identity

0

0

00

00matrix zero

:Examples

numbers ofarray ldimensiona twoa:Matrix


MATRICES

1000

1400

0140

3121

ngular upper tria,

451

501

112

symmetric

:Examples


Determinant of a MATRICES

82)015(1)512(1)25(2

50

1-31-

45

1-31-

45

502

451

501

132

det

:Examples

only matrices squarefor Defined


Adding and Multiplying Matrices

ji

ji

m

k

,bacBAC *

pm ifonly defined is ABCproduct The *

q)B(p and m)(nA matrices twooftion Multiplica

,bacBAC *

size same thehave they ifonly Defined *

B andA matrices twoofaddition The

1kjikij

ijijij


Systems of Linear Equations

formMatrix form Standard

7

5

3

601

315.2

342

76

535.2

3342

formsdifferent in

presented becan equationslinear of systemA

3

2

1

31

321

321

x

x

x

xx

xxx

xxx


Solutions of Linear Equations

52

3

:equations following theosolution t a is 2

1

21

21

2

1

xx

xx

x

x


Solutions of Linear Equations A set of equations is inconsistent if there

exists no solution to the system of equations:

ntinconsiste are equations These

542

32

21

21

xx

xx


Solutions of Linear Equations Some systems of equations may have infinite

number of solutions

allfor solution ais)3(5.0

solutions ofnumber infinite have

642

32

2

1

21

21

aa

a

x

x

xx

xx


Graphical Solution of Systems ofLinear Equations

52

3

21

21

xx

xx

solution


Cramer’s Rule is Not Practical

way efficient in computed are tsdeterminan theif used becan It

system. 30by 30 a solve toyears10 needscomputer super A

. systems largefor practicalnot is Rule sCramer'

2

21

11

51

31

,1

21

11

25

13

system thesolve toused becan Rule sCramer'

17

21 xx


Naive Gaussian Elimination Examples

Lecture 13 Naive Gaussian

Elimination


Naive Gaussian Elimination The method consists of two steps:

Forward Elimination: the system is reduced to upper triangular form. A sequence of elementary operations is used.

Backward Substitution: Solve the system starting from the last variable.

'

'

'00

''0

3

2

1

3

2

1

33

2322

131211

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

a

aa

aaa

b

b

b

x

x

x

aaa

aaa

aaa


Elementary Row Operations

Adding a multiple of one row to another

Multiply any row by a non-zero constant


ExampleForward Elimination

18

27

6

16

14320

18120

2240

4226

4 3, 2,equationsfrom Eliminate:Step1

nEliminatio Forward:1Part

34

19

26

16

18146

39133

106812

4226

4

3

2

1

1

4

3

2

1

x

x

x

x

x

x

x

x

x



3

9

6

16

3000

5200

2240

4226

4equation from Eliminate:Step3

21

9

6

16

13400

5200

2240

4226

4 ,3equationsfrom Eliminate:Step2

4

3

2

1

3

4

3

2

1

2

x

x

x

x

x

x

x

x

x

x



3

9

6

16

3000

5200

2240

4226

34

19

26

16

18146

39133

106812

4226

:nEliminatio Forward theofSummary

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x


ExampleBackward Substitution

36

)1(4)2(2)1(216,1

4

)1(2)2(26

22

59,1

3

3

for solve ,...for solvethen ,for Solve

3

9

6

16

3000

5200

2240

4226

12

34

134

4

3

2

1

xx

xx

xxx

x

x

x

x


Forward Elimination

ni

ba

abb

njaa

aaa

x

ni

ba

abb

njaa

aaa

x

ijj

ji

ijij

ijj

ji

ijij

3

)2(

eliminate To

2

)1(

eliminate To

222

2

222

2

2

111

1

111

1

1


Forward Elimination

.eliminated is until Continue

1

)(

eliminate To

1

n

mmm

imjj

mjmm

imijij

m

x

nim

ba

abb

njmaa

aaa

x


Backward Substitution

mm

n

mjjjmm

m

nn

nnnnnnnn

nn

nnnnn

nn

nn

a

xab

x

a

xaxabx

a

xabx

a

bx

,

1,

2,2

11,2,222

1,1

,111


Summary of the Naive Gaussian Elimination Example How to check a solution Problems with Naive Gaussian Elimination

Failure due to zero pivot element Error

Lecture 14Naive Gaussian

Elimination


Naive Gaussian Elimination

o The method consists of two stepso Forward Elimination: the system is reduced to

upper triangular form. A sequence of elementary operations is used.

o Backward Substitution: Solve the system starting from the last variable. Solve for xn ,xn-1,…x1.

'

'

'00

''0

3

2

1

3

2

1

33

2322

131211

3

2

1

3

2

1

333231

232221

131211

b

b

b

x

x

x

a

aa

aaa

b

b

b

x

x

x

aaa

aaa

aaa


Example 1

1775

64

832

11

333723

11

22210232

)(1832

3 ,2equations from Eliminate:Step1___nEliminatio Forward:1Part

:nEliminatioGaussian Naive using Solve

32

32

321

321

321

321

1

xx

xx

xxx

eqeqeqxxx

eqeqeqxxx

equationpivotunchangedeqxxx

x


Example 1

1313

64

832

21

5331775

)(264

1832

3equationfrom Eliminate:Step2nEliminatio Forward:1Part

3

32

321

32

32

321

2

x

xx

xxx

eqeqeqxx

equationpivotunchangedeqxx

unchangedeqxxx

x


Example 1Backward Substitution

1

2

1

issolution The

1328

21

46

113

13

3

2

1

1,1

32

1,1

33,122,111

3

2,2

33,222

3,3

33

x

x

x

a

xx

a

xaxabx

x

a

xabx

a

bx


How Do We Know If a Solution is Good or Not Given AX=B

X is a solution if AX-B=0 Due to computation error AX-B may not be zero Compute the residuals R=|AX-B|

One possible test is ?????

ii

rmax if acceptable issolution The


Determinant

13detdet

1300

410

321

A'

213

232

321

A

:Example

tdeterminan affect thenot do operations elementary The

operations Elementary

(A')(A)


How Many Solutions Does a System of Equations AX=B Have?

0 elements0 elements

B ingcorrespondB ingcorrespond

rows zerorows zero

moreor one has moreor one hasrows zero no has

matrix reducedmatrix reducedmatrix reduced

0det(A)0det(A)0det(A)

Infinitesolution NoUnique


Examples

5.1

!105.0

0

#:

0

2

00

21

1

2

00

21

1

1

20

21

4

2

42

21

3

2

42

21

2

1

43

21

solutions of # infintesolution NoUnique

XimpossibleX

solutionsInfinitesolutionNosolution

XXX

XXX


Lectures 15-16:Gaussian Elimination

with Scaled Partial Pivoting

Problems with Naive Gaussian Elimination Definitions and Initial step Forward Elimination Backward substitution Example


Problems with Naive Gaussian Elimination

o The Naive Gaussian Elimination may fail for very simple cases. (The pivoting element is zero).

o Very small pivoting element may result in serious computation errors

2

1

11

10

2

1

x

x

2

1

11

110

2

110

x

x


Example 2

1

1

1

1

3524

3685

4123

1211

:Pivoting Partial Scaledwith

nEliminatioGaussian using sytstem following theSolve

4

3

2

1

x

x

x

x


Example 2Initialization step

4321LVectorIndex

5842S vectorScale

1

1

1

1

3524

3685

4123

1211

4

3

2

1

x

x

x

xScale vector:

disregard sign

find largest in magnitude in each row


Why Index Vector? Index vectors are used because it is much

easier to exchange a single index element compared to exchanging the values of a complete row.

In practical problems with very large N, exchanging the contents of rows may not be practical since they could be stored at different locations.


Example 2Forward Elimination-- Step 1: eliminate x1

]1324[

Exchangeequation pivot first theis 4equation

toscorrespondmax 5

4,

8

5,

4

3,

2

14,3,2,1

]4321[

]5842[

1

1

1

1

3524

3685

4123

1211

equationpivot theofSelection

14

4

1,

4

3

2

1

L

landl

liS

aRatios

L

S

x

x

x

x

i

i

l

l



1

25.2

75.1

25.1

3524

75.025.05.50

75.175.25.00

25.075.05.10

1

1

1

1

3524

3685

4123

1211

B andA Update

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x

First pivot equation



]2314[

2

5.1

8

5.5

4

5.0:Ratios

1

25.2

75.1

25.1

3524

75.025.05.50

75.175.25.00

25.075.05.10

equationpivot second theofSelection

4

3

2

1

L

x

x

x

x



]2314[2

5.1

8

5.5

4

5.04,3,2:Ratios

]1324[]5842[

1

25.2

75.1

25.1

3524

75.025.05.50

75.175.25.00

25.075.05.10


2,

4

3

2

1

LiS

a

LS

x

x

x

x

i

i

l

l



1

9

1667.2

25.1

3524

2000

8333.15.200

25.075.05.10

]3214[

1

8333.6

1667.2

25.1

3524

6667.125.000

8333.15.200

25.075.05.10

4

3

2

1

4

3

2

1

x

x

x

x

L

x

x

x

xThird pivot equation



7.23334

2531

1.13335.1

75.025.025.1

2.43275.2

8333.11667.2,5.4

2

9

]3214[

1

9

1667.2

25.1

3524

2000

8333.15.200

25.075.05.10

234

1,

22,33,44,1

34

2,

33,44,2

4

3,

44,3

4,4

4

3

2

1

1

1111

2

222

3

33

4

4

xxx

a

xaxaxabx

xx

a

xaxabx

x

a

xabx

a

bx

L

x

x

x

x

l

llll

l

lll

l

ll

l

l


Example 3

1

1

1

1

3524

3685

4123

1211

Pivoting Partial Scaledwith

nEliminatioGaussian using sytstem following theSolve

4

3

2

1

x

x

x

x


Example 3Initialization step

4321LVectorIndex

5842S vectorScale

1

1

1

1

3524

3685

4123

1211

4

3

2

1

x

x

x

x



]1324[

Exchangeequation pivot first theis 4equation

toscorrespondmax 5

4,

8

5,

4

3,

2

14,3,2,1

]4321[

]5842[

1

1

1

1

3524

3685

4123

1211

equationpivot theofSelection

14

41,

4

3

2

1

L

landl

liS

aRatios

L

S

x

x

x

x

i

i

l

l



1

25.2

75.1

25.1

3524

75.025.05.100

75.175.25.00

25.075.05.10

1

1

1

1

3524

3685

4133

1211

B andA Update

4

3

2

1

4

3

2

1

x

x

x

x

x

x

x

x



]1234[

2

5.1

8

5.10

4

5.0

2

5.1

8

5.10

4

5.0:Ratios

1

25.2

75.1

25.1

3524

75.025.05.100

75.175.25.00

25.075.05.10


4

3

2

1

L

x

x

x

x



]1234[2

5.1

8

5.10

4

5.04,3,2:Ratios

]1324[]5842[

1

25.2

75.1

25.1

3524

75.025.05.100

75.175.25.00

25.075.05.10


2,

4

3

2

1

LiS

a

LS

x

x

x

x

i

i

l

l



1

2.25

1.8571

0.9286

3524

75.025.05.100

1.71432.7619-00

0.35710.785700

]2314[

1

25.2

75.1

25.1

3524

75.025.05.100

75.175.25.00

25.075.05.10

B andA Updating

4

3

2

1

4

3

2

1

x

x

x

x

L

x

x

x

x



]1234[2

0.7857

4

2.76194,3:Ratios

]1234[]5842[

1

2.25

1.8571

0.9286

3524

75.025.05.100

1.71432.761900

0.35710.785700

equationpivot third theofSelection

3,

4

3

2

1

LiS

a

LS

x

x

x

x

i

i

l

l



1

2.25

1.8571

1.4569

3524

75.025.05.100

1.71432.761900

0.8448000

]1234[

1

2.25

1.8571

0.9286

3524

75.025.05.100

1.71432.761900

0.35710.785700

4

3

2

1

4

3

2

1

x

x

x

x

L

x

x

x

x



1.86734

2531

0.3469

0.39802.7619

1.71431.8571,1.7245

0.8448

1.4569

]1234[

1

2.25

1.8571

1.4569

3524

75.025.05.100

1.71432.761900

0.8448000

234

1,

22,33,44,1

2,

33,44,2

4

3,

44,3

4,4

4

3

2

1

1

1111

2

222

3

33

4

4

xxx

a

xaxaxabx

a

xaxabx

x

a

xabx

a

bx

L

x

x

x

x

l

llll

l

lll

l

ll

l

l


How Good is the Solution?

0.001

0.003

0.002

0.005

:Residues

1.7245

0.3980

0.3469

1.8673

1

1

1

1

3524

3685

4123

1211

4

3

2

1

4

3

2

1

R

x

x

x

x

solution

x

x

x

x


Remarks: We use index vector to avoid the need to move

the rows which may not be practical for large problems.

If we order the equation as in the last value of the index vector, we have a triangular form.

Scale vector is formed by taking maximum in magnitude in each row.

Scale vector do not change. The original matrices A and B are used in

checking the residuals.


Lecture 17 Tridiagonal & Banded

Systems and Gauss-Jordan

Method

Tridiagonal Systems Diagonal Dominance Tridiagonal Algorithm Examples Gauss-Jordan Algorithm


Tridiagonal Systems: The non-zero elements are

in the main diagonal, super diagonal and subdiagonal.

aij=0 if |i-j| > 1

5

4

3

2

1

5

4

3

2

1

61000

14100

02620

00143

00015

b

b

b

b

b

x

x

x

x

x

Tridiagonal Systems


Occur in many applications Needs less storage (4n-2 compared to n2 +n for the general cases)

Selection of pivoting rows is unnecessary (under some conditions)

Efficiently solved by Gaussian elimination

Tridiagonal Systems


Based on Naive Gaussian elimination. As in previous Gaussian elimination algorithms

Forward elimination step Backward substitution step

Elements in the super diagonal are not affected. Elements in the main diagonal, and B need

updating

Algorithm to Solve Tridiagonal Systems


Tridiagonal System

nnn

n

nnnn

n

b

b

b

b

x

x

x

x

d

c

d

cd

cd

b

b

b

b

x

x

x

x

da

c

da

cda

cd

b a, d

3

2

1

3

2

1

1

3

22

11

3

2

1

3

2

1

1

1

32

221

11

and and d update toneed We


Diagonal Dominance

row. ingcorrespond in the elements of sum the

nlarger tha iselement diagonaleach of magnitude The

)1(aa

ifdominant diagonally is matrix A

,1ijii nifor

An

ijj


Diagonal Dominance

dominant DiagonallyNot dominant Diagonally

121

232

103

521

161

103

:Examples


Diagonally Dominant Tridiagonal System

A tridiagonal system is diagonally dominant if

Forward Elimination preserves diagonal dominance

)1(1 niacd iii


Solving Tridiagonal System

1,...,2,1for 1

onSubstituti Backward

2

nEliminatio Forward

1

11

1

11

1

nnixcbd

x

d

bx

nibd

abb

cd

add

iiii

i

n

nn

ii

iii

ii

iii


Example

1,2,3for 1

,

onSubstituti Backward

42,

nEliminatio Forward

6

8

9

12

,2

2

2

,1

1

1

,

5

5

5

5

6

8

9

12

51

251

251

25

Solve

1

11

11

1

1

4

3

2

1

ixcbd

xd

bx

ibd

abbc

d

add

BCAD

x

x

x

x

iiii

in

nn

ii

iiii

i

iii


Example

5619.45652.4

5652.616,5619.4

5652.4

215

5652.66.4

6.618,5652.4

6.4

215

6.65

1219,6.4

5

215

nEliminatio Forward

6

8

9

12

,2

2

2

,1

1

1

,

5

5

5

5

33

3443

3

344

22

2332

2

233

11

1221

1

122

bd

abbc

d

add

bd

abbc

d

add

bd

abbc

d

add

BCAD


ExampleBackward Substitution

After the Forward Elimination:

Backward Substitution:

25

1212

16.4

126.6

15652.4

125652.6

,15619.4

5619.4

5619.45652.66.612,5619.45652.46.45

1

2111

2

3222

3

4333

4

44

d

xcbx

d

xcbx

d

xcbx

d

bx

BD TT


Gauss-Jordan Method The method reduces the general system of

equations AX=B to IX=B where I is an identity matrix.

Only Forward elimination is done and no substitution is needed.

It has the same problems as Naive Gaussian elimination and can be modified to do partial scaled pivoting.

It takes 50% more time than Naive Gaussian method.


Gauss-Jordan MethodExample

2

7

0

200

560

111

11

233

11

422

2/11

32fromxEleminate1

2

7

0

422

124

222

3

2

1

1

3

2

1

x

x

x

eqeqeq

eqeqeq

eqeq

andequationsStep

x

x

x



2

1.1667

1.1667

200

0.833310

1667.001

21

033

21

111

6/22

31fromxEleminate2

2

7

0

200

560

111

3

2

1

2

3

2

1

x

x

x

eqeqeq

eqeqeq

eqeq

andequationsStep

x

x

x



1

2

1

100

010

001

31

0.833322

31

1667.011

2/33

21fromxEleminate3

2

1.1667

1.1667

200

0.833310

1667.001

3

2

1

3

3

2

1

x

x

x

eqeqeq

eqeqeq

eqeq

andequationsStep

x

x

x



1

2

1

1

2

1

100

010

001

2

7

0

422

124

222

3

2

1

3

2

1

3

2

1

x

x

x

issolution

x

x

x

todtransformeis

x

x

x

cise301_topic3kfupm1 se301: numerical methods topic 3: solution of systems of linear equations...

Documents

practical slide

textbook slide

constant slide

systems of equations

system of equations

steps oforward elimination

linear equations lectures

set of equations