ddis slides 1

8/10/2019 Ddis Slides 1

1/129

Large Linear system resolution, domain decomposition anditerative solver PART 1

Nicolas Chevaugeon

GeM - Institut de Recherche en G enie Civil et M ecaniqueUMR CNRS - Ecole Centrale Nantes- Universit e de Nantes


2/129

Introduction Matrix Storage Matrix Multiplication Basic Linear Algebra and Matrix properties

Section 1

Introduction

2 / 85


3/129


This class is dedicated to the Resolution of large system of equation.Such system arise in most computational mechanics applications.Typically one has to solve Ku= fWe will look at some details, related to the computer implementationof this problem, focussing on

Efficiency,

Problems related to the size and the memory managment ...

Accuracy.

Parallelism.

Sparcity of system

Direct/Iterative Solvers

3 / 85


4/129


Layout

introduction, motivation

matrix storage

efficient basic linear algebra operation : blas

matrix properties

dense matrix factorisation and system resolution

sparse direct solver

iterative solver

preconditioners

domain decomposition methods

multigrid (Jeroen Wackers)

4 / 85


5/129


Recommended books

Matrix Computations 1996, by Gene H. Golub , Charles F. van

Van Loan. Iterative Methods for Sparse Linear Systems, by Yousef Saad.

Numerical methods in computational mechanics M. Okrouhlik.

5 / 85

I t d ti M t i St M t i M lti li ti B i Li Al b d M t i ti


6/129


Introduction

size of problem. compared to memory of typical computer.

Most numerical method for pde : transform the pde into a set of

discrete non linear equation which are the linearised to be solvediteratively.

each iteration : a linear system to solve. Size scale with numberof dof.

number of dof in 3D :(L/h)3

6 / 85



7/129


Memory size

size size 2 memory usage

1000 1.e6 7.6 Mb

10000 1.e8 762.94 Mb20000 4.e8 3. Gb40000 16.e8 12. Gb

Table : Memory usage of a nxn matrix using double precision

7 / 85



8/129


Many applications give rise to (non) linear systems of equations.Typically the system matrices are :

Large , 108 unknowns are not exceptional anymore;

sparse, only a fraction of the entries of the matrix is nonzero;

structured : the matrix is often symmetric or of symmetric paternand is banded.

The Matrix have special numerical properties eg : positive definite,Diagonal dominant ...

8 / 85



9/129


Traditionally there are 2 classes of methods :

direct method, usually a factorisation of the matrix These methodare exact up to truncature error.

iterative method : a series of approximated solution isconstructed, using simple operation like matrix vector product.

For Many problem it is not clear what method is best between direct

or iteratives solver. Direct methods are robust. But can need (very) large memory,

and may take to long for some application

Iterative methods may not converge, but if low precision only isneeded, can be much faster than direct method.

A combinaison of direct and iterative may be used domaindecomposition, iterative refinment.

9 / 85



10/129


Does the best iterative method exist ?

In general : No. For certain class of matrix : yes. Succes of iterativemethods depends on :

properties ofA () Size, sparsity structure, symmetry,

conditionning ..)

Computer Architecture (scalar, parallel, memory organisation ...)

Memory space available.

Accuracy required.

10 / 85



11/129

g p g p p

Section 2

Matrix Storage

11 / 85



12/129

g p g p p

Storing a Matrix seems like something simple ... Its just an array withtwo dimension !

12 / 85



13/129

Storing a Matrix seems like something simple ... Its just an array withtwo dimension !Well, in fact all that can really be stored in computer are data inmemory address. A system can reserve contiguous memoryaddress, to mimic a 1 dimensional array.So how to store a 2d array in what is basically a 1d array ?

12 / 85



14/129

Storing a Matrix seems like something simple ... Its just an array withtwo dimension !Well, in fact all that can really be stored in computer are data inmemory address. A system can reserve contiguous memoryaddress, to mimic a 1 dimensional array.So how to store a 2d array in what is basically a 1d array ?What if storing all the entry of a matrix is not practicable and or

consume to much space ?

12 / 85



15/129

Storing a Matrix seems like something simple ... Its just an array withtwo dimension !Well, in fact all that can really be stored in computer are data inmemory address. A system can reserve contiguous memoryaddress, to mimic a 1 dimensional array.So how to store a 2d array in what is basically a 1d array ?What if storing all the entry of a matrix is not practicable and or

consume to much space ?Most matrix comming from numerical methods such as finite element,finite difference or finte volume are very sparse. Which means thatthe majority of the terms are equal to zero. Why store them then ?

12 / 85



16/129

Storing a Matrix seems like something simple ... Its just an array withtwo dimension !

Well, in fact all that can really be stored in computer are data inmemory address. A system can reserve contiguous memoryaddress, to mimic a 1 dimensional array.So how to store a 2d array in what is basically a 1d array ?What if storing all the entry of a matrix is not practicable and or

consume to much space ?Most matrix comming from numerical methods such as finite element,finite difference or finte volume are very sparse. Which means thatthe majority of the terms are equal to zero. Why store them then ?These simple observations leads to a variety of matrix storage

scheme that any numericists need to know of : They affect the waysimple operation like Matrix vector product must be written efficiently,how to design a solver, etc ...

12 / 85



17/129

outline

Full storage : Row major, column major

Symetric storage

Band storage Skyline storage

Coordinate Storage

Compressed row storage/ column storage

13 / 85



18/129

full row major storage

A=

a1,1 a1,2 a1,na2,1 a2,2 a2,n

.

.....

an,1 an,2 an,n

can be strored in an arrayvalin the full storage format by row. This iscalled therow major format.

k 1 2

n n+1

2

n

(n

1)

n+1

n

n

val a 1,1 a1,2 a1,n a2,1 a2,n an,1 an,n

14 / 85



19/129

full row major storage

We thus have :

ai,j=val[(i 1) n+j] val[k] =a((k1)/n)+1,((k1)%n+1)

We are using here the so calledfortranconvention, where array indexstart at 1. Incindexing typically start at 0. be carefull when usinga library ... Which convention is used ??

15 / 85



20/129

full column major storage

The matrixA can be strored in an arrayvalin the full storage formatby column. This is called the column major format.

k 1 2

n n+1

2

n

(n

1)

n+1

n

n

val a 1,1 a2,1 an,1 a1,2 an,2 a1,n an,n ai,j=val[(i+ (j 1) n] val[k] =a((k1)%n+1),((k1)/n)+1

16 / 85



21/129

row major or column major format are widely used, in particular fordense linear algebra. Blas and lapack libraries in particular expect acolumn major format. Of course this format can also be used to store

rectangular matrix. The storage space used is typicallyNotion ofLeading direction ...

17 / 85



22/129

triangular storage

Triangular matrix or symmetric matrix are typically stored usingtriangular storage. It is similar to the full storage but only n(n+1)/2terms are stored, which mean roughly half of the memory is saved.Example :

A=

a1,1 a1,2 a1,3a1,2 a2,2 a2,3

a1,3 a2,3 a3,3

k 1 2 3 4 5 6

val a 1,1 a1,2 a1,3 a2,2 a2,3 a3,3

18 / 85



23/129

A typical sparse system

Finite Element Analysis, 2d, number of nodes = 1128. 2208 DOFS.

nnz = 30118. sparsity: 0.06%

19 / 85



24/129

A typical sparse system

Before Renumbering :

bandwidth 2159Full storage 4.6 e6 doublesFull storage sym 2.4e6 doublesBand storage 9.5 e6 doublesSym Band storage 4.7 e6 doubles

20 / 85



25/129

renumbering

After renumbering using reverse cutHill McKee :

bandwidth 178Full storage 4.6 e6 doublesFull storage sym 2.4e6 doublesBand storage 7.8 e5 doublesSym Band storage 3.9 e5 doubles

21 / 85



26/129

band storage

Band storage depends on two value : the number of non-zero lower

subdiagonalkland the number of superdiagonalku.the bandwith isku+kl+1Example of a 5x5 matrix, withku=1 andkl=2

A=

a1,1 a1,2 0 0 0a2,1 a2,2 a2,3 0 0a3,1 a3,2 a3,3 a3,4 0

0 a4,2 a4,3 a4,4 a4,50 0 a5,3 a5,4 a5,5

Band storage : m n ku kl

5 5 1 2k 1 2 3 4 5 6 7 8 9 10

val a1,2 a2,3 a3,4 a4,5 a1,1 a2,2 a3,3 a4,4 a5,5k 11 12 13 14 15 16 17 18 19 20

val a 2,1 a3,2 a4,3 a5,4 a3,1 a4,2 a5,3 22 / 85



27/129

band storage

A matrix in band storage take the space of(kl+ku+1)xn

23 / 85



28/129

skyline storage

Usefull for skyline matrices, also called variable band or profilematrices.

Mostly used for symmetric matrix. Variant exist for

non-symmetric. The matrix element are stored column by column, each column

beginning with the first non zero-element and ending with thediagonal element. An array col ptr store the beginning of column

j.

Very popular for direct solver for medium sized system.

24 / 85



29/129

skyline storage

Example:

A=

a1,1 0 a1,3 0 00 a2,2 0 a2,4 0

a1,3 0 a3,3 a3,4 00 a2,4 a3,4 a4,4 a4,50 0 0 a4,5 a5,5

k 1 2 3 4 5 6 7 8 9 10

val a 1,1 a2,2 a1,3 0 a3,3 a2,4 a3,4 a4,4 a4,5 a5,5

j 1 2 3 4 5 6

col ptr(j) 1 2 3 6 9 11

25 / 85



30/129

If the matrix is sparse, but no good structure can appear, we need aspecial storage scheme so that only the non-zero terms are stored

26 / 85



31/129

Coordinate Storage

The simplest approach to store sparse matrix.the values of the non-zero elements are stored with their coordinate,in 3 separates arrays : val, row idx, col idx. The order usally do notmatter.

A=

a1,1 a1,2 0 a1,4 0

0 a2,2 a2,3 a2,4 00 0 a3,3 0 a3,5

a4,1 0 0 a4,4 a4,50 a5,2 0 0 a5,5

k 1 2 3 4 5 6 7 8 9 10 11 12 13

val(k) a 41 a35 a33 a52 a11 a23 a55 a24 a12 a44 a14 a45 a22row index(k) 4 3 3 5 1 2 5 2 1 4 1 4 2

col idx(k) 1 5 3 2 1 3 5 4 2 4 4 5 2

27 / 85


C d


32/129

Compressed row storage

A=

a 0 b 0 00 0 c d 00 e 0 f 0g h 0 0 ij 0 k 0 0

This matrix as 11 non zero term (number of non zero,nnz=11).We store the entry row by row in the valarray of dimensionnnzThe array col index ofnnzinteger store the column index of thecorresponding entry.

28 / 85


C d


33/129


A=

a 0 b 0 00 0 c d 00 e 0 f 0g h 0 0 ij 0 k 0 0

k 1 2 3 4 5 6 7 8 9 10 11

val(k) a b c d e f g h i j k

col index(k) 1 3 3 4 2 5 1 2 5 1 3

28 / 85


C d t


34/129


A=

a 0 b 0 00 0 c d 00 e 0 f 0g h 0 0 ij 0 k 0 0

k 1 2 3 4 5 6 7 8 9 10 11


col index(k) 1 3 3 4 2 5 1 2 5 1 3

The Array row ptr(i) is a array of integer of size n+1 (wherenis the

number of line of the matrix) that contain the index of the start of line(i) in array val, for all entry but the last that contain nnz+1

28 / 85


C d t


35/129


A=

a 0 b 0 00 0 c d 00 e 0 f 0g h 0 0 ij 0 k 0 0

k 1 2 3 4 5 6 7 8 9 10 11


col index(k) 1 3 3 4 2 5 1 2 5 1 3

i 1 2 3 4 5 6

row ptr(i) 1 3 5 7 10 12

28 / 85



36/129

How to find entry (i,j) of a matrixA in compressed row storage ?

29 / 85



37/129


Find the begin and end index of line i :kb= row ptr(i),ke=row ptr(i+1) 1

29 / 85


38/129



39/129



search forj in col index(kb :ke)

ifjnot found A(i,j) =0

29 / 85



40/129



search forj in col index(kb :ke)

ifjnot found A(i,j) =0 ifjfound at indexkj in(kb, ke), A(i,j) =val(kj)

29 / 85



41/129

We just cover most of the matrix storage scheme.Other scheme exist.In particular, block variant of the previous schemes are also used.

30 / 85



42/129

Section 3

Matrix Multiplication

31 / 85


Basic Dense Matrix Multiplication


43/129


A Rm,p

B Rp,n

C Rm,n

C= AB

32 / 85




44/129


A Rm,p B Rp,n C Rm,nC= AB

matrix multiplication algorithm :

for i=1to mdofor j=1 tondo

for k=1 topdoC(i,j) = A(i,k)*B(k,j) +C(i,j)

end

end

end

32 / 85




45/129


A Rm,p B Rp,n C Rm,nC= AB

one variant, reordering the loop :

for j=1to ndofor i=1 tomdo

for k=1 topdoC(i,j) = A(i,k)*B(k,j) +C(i,j)

end

end

end

32 / 85




46/129


One can construct easily 6 variants : ijk jik ikj jki kij kjiDoes it matter ?

33 / 85




47/129


One can construct easily 6 variants : ijk jik ikj jki kij kjiDoes it matter ?Well lets mesure.Pick a matrix from Matrix Market

http://math.nist.gov/MatrixMarket/

Matrix size : 1806X1806.number of floating point operations : 18063 2= 11, 8e9.Note that the number of operations is the same whatever thealgorithm.

33 / 85




48/129

p

One can construct easily 6 variants : ijk jik ikj jki kij kjiDoes it matter ?Well lets mesure.Pick a matrix from Matrix Market

http://math.nist.gov/MatrixMarket/

Matrix size : 1806X1806.number of floating point operations : 18063 2= 11, 8e9.Note that the number of operations is the same whatever thealgorithm.

algorithm ijk jik ikj jki kij kjitime (s) 19.79 28.47 50.63 9.82 53.57 25.23

33 / 85


Cache effect


49/129

What changed from one version to another is the time it take toaccess and modify the data.

34 / 85


Cache effect


50/129


When something has to be computed, it has to be moved from themain memory to the register, up and down the memory hierachy.

34 / 85


Cache effect


51/129


When something has to be computed, it has to be moved from themain memory to the register, up and down the memory hierachy.

The running time of the loops are dominated by the memory accessto the array, usually not by the operation them-self.

34 / 85


Cache effect


52/129

CPU do not access memory byte to byte, Instead they fetchmemory chunks called cache lines (typically 64 bytes.).

35 / 85


Cache effect


53/129


When a memory location is read, an entire cache line is movedto the cache. Thats the most time consuming operation.

35 / 85


Cache effect


54/129



If for the next instruction, every data are already in the cache,nothing need to be fetch it goes much faster.

35 / 85


Cache effect


55/129



If for the next instruction, every data are already in the cache,nothing need to be fetch it goes much faster.

In the matrix multiplication example, The Matrix were stored incolumn major format. It means, that the inner loop would gomuch faster if during this loop the next element computed is on

the next line of each matrix and therefore already in the cache.

35 / 85


Of course that not the only factor. One can look at


56/129

Processor level parallelism permit to performs more that oneoperation at once if evrey things are ready in the registery.

Pipelining (chaining operation like on an assembly line) atprocessor level

On multicore processor, The operation can be parralleze atthread level

..

36 / 85




57/129




..

LESSON : Even for something as simple as matrix multiplication itcan be very difficult to do it in the most efficient way. Typical goodimplementation will work on matrix block having the perfect size to fitin cache memory, taking advantage of piplining and all level ofparallelism.

36 / 85




58/129




..

LESSON : Even for something as simple as matrix multiplication itcan be very difficult to do it in the most efficient way. Typical goodimplementation will work on matrix block having the perfect size to fitin cache memory, taking advantage of piplining and all level ofparallelism.

Computational kernel, that are the most used in any application needto be fine tuned to obtain good performance. Its of course the case ofbasic linear algebra computation. One has to use well programmedlibrary to move faster. One good example is the BLAS Library.

36 / 85


BLAS : basic linear Algebra Subroutine


59/129

BLAS define an interface for some basic linear algebra algorithm

such as

vector copying, scalling, adding

vector dot products, norms.

Matrix vector products

Matrix matrix product Triangular matrix solve.

This interface is mostly defined in fortran but can be called from anylanguage. The interface was first published in 1979, And is usedsince as a building block in higher level math programming languages

and libraries (matlab; numpy, lapack ...).BLAS subroutines are a standard API (Application programminginterface.)

37 / 85




60/129

BLAS libraryimplementationexists, new version come all the time,specially tuned for new architecture.

The reference and most straight forward implementation is theone from netlib (www.netlib.org/blas). (opensource)

38 / 85




61/129



ATLAS Automatically Tuned Linear Algebra Software.

38 / 85




62/129




OpenBLAS Optimized blas based on GOTO blas

38 / 85




63/129





clBLAS : OpenCL implementation of BLAS. OpenCL is thedevelopping standard to framework to write program accrossheterogeneous framework (CPU-GPU mixte)

38 / 85




64/129





clBLAS : OpenCL implementation of BLAS. OpenCL is thedevelopping standard to framework to write program accrossheterogeneous framework (CPU-GPU mixte)

cuBLAS Optimized for NVIDIA GPU cards (cuda) ...

38 / 85




65/129

Each processor vendor have his own BLAS implementation tuned forthere processor :

ACML : part of the AMD Core MATH Library : AMD Athlon andopteron

39 / 85




66/129



ESSL : IBMs Engineering and Scientific Subroutine Library,

PowerPC

39 / 85




67/129




PowerPC

HP MLIB support IA-64 PA-RISC, x86 OPTERON under HPUXand linux

39 / 85




68/129




PowerPC HP MLIB support IA-64 PA-RISC, x86 OPTERON under HPUX

and linux

intel MKL : Intel Math Kernel Library. Optimization for Intelprocessor

39 / 85




69/129




PowerPC HP MLIB support IA-64 PA-RISC, x86 OPTERON under HPUX

and linux

intel MKL : Intel Math Kernel Library. Optimization for Intelprocessor

...

39 / 85



70/129

Most linear algebra algorithm can be split into call to BLASsubroutine.By doing so, a programmer rely on the BLAS implementation to dothe heavy lifting, concentrating on his own algorithm.Changing the BLAS implementation to get faster result is just then amatter of linking to the new implementation. Nothing need to bechanged in the user code.

40 / 85


BLAS overview


71/129

Each routine follow the following name convention : first letter is eithers or d, for single or double precision version of the routine.BLAS is divided in 3 Level.

41 / 85


BLAS overview


72/129

Level 1 : Vector operation

xSWAP exchange the values in 2 array

xSCAL scal one vector

xCOPY copy one vector

xAXPY addxto y

xDOT compute the dot product of 2 vector

xNRM2 Compute the 2-norm on 1 vector

...

41 / 85


73/129


BLAS overview


74/129

Level 3 : Matrix Matrix operation

xGEMM general matrix matrix product

xSYMM symetric matrix matrix product

xSYRK Rank k update C = C + AAT

xSYRK2 Rank k updateC = C + BAT

xTRMM triangular matrix - matrix product

xTRSM Triangular matrix solve with matrix right hand side.

41 / 85



75/129

BLAS function are very flexible, while behing compatible with mostlanguage. This flexibility come at the price of the number of argument

to be pass to each subroutine.For example AXPY that perform the operationy = x +yneed 6arguments :

xAXPY(N, ALPHA, X, INCX, Y, INCY)

xGEMM That perform the operationC = AB + Cneeds 13arguments :

xGEMM(TRANSA, TRANSB, M, N, K, ALPHA, A, LDA,

B, LDB, BETA, C, LDC)

42 / 85


back to mesurement


76/129

Same matrix as in the first example. Computation time for theproduct.

best hand made loop : 9.81 s

Using netlib blas : 4.56 s

Using open blas : 1.4 s

43 / 85


Blas Benchmark


77/129

44 / 85


Blas Benchmark


78/129

Lesson : Whenever possible, formulate algorithm in terms of Level 3blas operation.

44 / 85



79/129

Lots of algorithm can be recast in matrix multiplication For example,integration of a bilinear form over an element :

Mij=

e

Ni(x)Nj(x)d

=ngauss

k=1

wkNi(xk)Nj(xk)J(xk)

45 / 85


80/129



81/129

For a given element, Compute the weighted jacobian wkJk=wkJ(xk)at each gauss point.The mass matrix can now be computed as :

M= NTdiag(w1J1, , wngaussJngauss)N

Using matrix Algebra and therefore a BLAS implementation, will ingeneral be much faster than the direct loop.

47 / 85


Sparse BLAS ??


82/129

Unfortunetely, this does not exist (yet?)Some implementation of BLAS do offer some support.

Efficient Libraries for sparse linear algebra, use complex algorithm topack multiple operations in BLAS call.

48 / 85


83/129


Space, subspace, span

A set of n vectors


84/129

A set ofnvectors

{a1, , an} in Rm

is linearly independent ifn

j=1

jaj=0 impliesj=0 j Otherwise, the set is linearly dependent.

A subspace of Rm is a subset that is also a vector space. Given a collection of vector {a1, , an} in Rm, the set of all

linear combinaisons of these vectors is a subspace called thespanof {a1, , an}

span {a1, , an} =

nj=1

jaj :j R

50 / 85


85/129


86/129


Matrix transpose


87/129

Given a matrixA in Rm,n, its transpose, notedAT is the matrix inRn,m such as

yAx= xATy

x R

n

y

Rm (1)

and we have:Aij=A

Tji

53 / 85 Introduction Matrix Storage Matrix Multiplication Basic Linear Algebra and Matrix properties

Symetric matrix


88/129

The following proposition are equivalent :

A matrixS is symetric if(Sx)T =xTS x. A matrixS is symetric ifS

T

=S A matrixS is symetric ifsij=sji

54 / 85


89/129


determinant


90/129

A recursive definition of the determinant of a n, nmatrix:

if A R1,1,det(A) =a11

else det(A) =

n

j=1

(1)j+1det(A1,j)

whereA1,j Rn1,n1 is obtained by removing the firstrow and thejcolumn fromA

A1,j=A((2, , n), (1, ,j 1,j+1, n))Ais singular if det(A) =0 otherwiseA is regular.

56 / 85


determinant


91/129

Properties of the determinant :

det(AB) =det(BA)

det(AT) =det(A)

det(A) =ndet(A)

det(I) =1

Characteristic polynomial : pA() =det(A I)

57 / 85


Identity Matrix


92/129

The following proposition are equivalent :

the matrixI is the identity matrix Ix = x x the matrixI is the identity matrix if Iij=ij

whereij

is the kroenecker symbol :

ii=1 (2)

ij=0 if i=j (3)

58 / 85


93/129


Positive Matrix


94/129

A matrixA is positive definite ifxTAx> 0 x=0

60 / 85


Positive Matrix


95/129

A matrixA is positive definite ifxTAx> 0 x=0symetric positive definite matrix are of special interest incomputational mechanics.

Factorisation for solving can be simplifyed.Special Iterative solver are available.

60 / 85


96/129


97/129


98/129


99/129


100/129


Singular Value Decomposition


101/129

ifA is realm nmatrix, then there exist orthogonal matricesU= [u1, , um] Rm,m and V= [v1, , vn] Rn,n

such that

UTAV= diag(1, , p) p=min{m, n}where1 2 p 0

Theiare called the singular values.Theuiare the Left singular vector.

Theviare the Right singular vector.

66 / 85




102/129

lets definer as :

1 r> r+1 = =p=0

ris the number of the last non-zero singular value. We the have :

rank(A) =r

null(A) =span{vr+1, , vn}ran(A) =span{u1, , ur}

67 / 85



l t d fi


103/129

lets definer as :

1 r> r+1 = =p=0

ris the number of the last non-zero singular value. We the have :

rank(A) =r

null(A) =span{vr+1, , vn}ran(A) =span{u1, , ur}

This allow to define quasi or numerically singular matrix: if singularvalue less than a fixed value, the matrix will be said numericallysingular.

67 / 85


vector norm


104/129

Vector norm : A vector normis an applicationf from Rn to R thatsatisfy the following properties :

f(x) 0 x Rn

f(x) =0 x= 0f(x +y) f(x) +f(y) (x), (y) Rn

f(x) = ||f(x) R, x Rn

68 / 85


vector norm


105/129

Notation : a norm i noted ||x||. Thep-norm is defined by :

||x||p

= (n

i=0 |

xi|

p)1p p

1

69 / 85


106/129


107/129


matrix norm


108/129

A matrix norm is an application from Rm,n to R With the followingproporties (the same as a vector norm) :

f(A) 0 A Rm,n

f(A) =0 A= 0f(A +B) f(A) +f(B) (A), (B) Rm,n

f(A) = ||f(A) R, A Rm,n

71 / 85


109/129


matrix norm


110/129

Submultiplicative propertie of thep-norm :

||AB

||p

||A

||p

||B

||p

73 / 85


matrix norm


111/129


||AB||p ||A||p||B||p

Note that it is not the case for all norms ! For instance, take||A||=max|aij|, andA = B=

1 11 1

. We then have

||AB|| > ||A||||B||

73 / 85


matrix norm


112/129


||AB||p ||A||p||B||pNote that it is not the case for all norms ! For instance, take

||A

||=max

|a

ij|, andA

=B

=1 1

1 1. We then have

||AB|| > ||A||||B||Important propertie of thep-norm :

||Ax||p ||A||p||x||p

73 / 85


some matrix norm properties

m


113/129

||A||1 = max1jn

i=1

|aij|

||A||= max1im

nj=1

|aij|

||A||1 is the maximum of the sum of each column.||A|| is the maximum of the sum of each line.1 and norms are easily computed.

||A(i1 :i2,j1 :j2)||p ||A||pThep-norm of a sub matrix is less or equal to the p-norm of thematrix.

74 / 85


The 2-norm is not as easy to compute. It can be shown that the the


114/129

2-norm ofA is its largest singular value1. (equal to the largesteigen value ofATA). In most case approximations can be used sincewe have, for instance :

1n||

A

||

||A

||2

m

||A

||

1m

||A||1 ||A||2

n||A||1

75 / 85


115/129


condition number : sensitivity of a linear problem

letx be the solution of Ax = b lets perturb the system by Fand f.


116/129

where is small.letx()be the solution of the perturbed system. We want to quantifythe variation of the solution of the system. (x() =x)We have

(A + F)x() =b + f

Lets differenciate with regard to.

Fx() + (A + B)x() =f

x()=0 =A1(f

Fx)

77 / 85



Lets develop a Taylor serie for x:

2


117/129

x() =x + x(0) +O( )

Now we can put bounds to the effect of the perturbation onx:

||x() x||

||x

|| ||x(0)||

||x

|| +O(2)

||||A1||( ||f||||x|| + ||F||) +O(2)

We haveAx = b, therefore 1||x|| ||A||||b|| and :

||x() x||||x|| ||A||||A

1||

|| ||F||||A|| + ||||f||||b||

+O(2)

78 / 85




118/129

(A) = ||A||||A1|| is called thecondition number of of(A)relative to a matrix norm ||.||

79 / 85




119/129

(A) = ||A||||A1|| is called thecondition number of of(A)relative to a matrix norm ||.||

The previous equation show that the relative error inx(solutionof the linear systemAx= b) can be as big as(A)times the

relative error onA and the relative error onb.

79 / 85


120/129


121/129


Eigen value decomposition

Eigenvalues are the roots of the characteristic polynomialp(z) =det(zI A)


122/129

81 / 85


123/129



Eigenvalues are the roots of the characteristic polynomialp(z) =det(zI A)Th f (A) 1 f h h i i l i l i


124/129

The set of roots(A) =1, , nof the characteristic polynomial isthespectrumofA.we have detA= 12 nandtr(A) =

ni=1

aii=i=1

i

81 / 85



Eigenvalues are the roots of the characteristic polynomialp(z) =det(zI A)Th t f t (A) 1 f th h t i ti l i l i


125/129

The set of roots(A) =1, , nof the characteristic polynomial isthespectrumofA.we have detA= 12 nandtr(A) =

ni=1

aii=i=1

i

The vectorsz, non zero that satisfy: Az= zare the eigenvectors.An eigenvector defines a one dimensional subspace that is invariantwith respect to pre-multiplication byA

81 / 85


eigen value decomposition

Schur decomposition ifAC

n,n there exists a unitaryQC

n,n suchh


126/129

C Cthat

QHAQ= T= D +N

whereD = diag(1, , n)and N Cn,n

strictly upper triangular.ifA Rn is diagonalisable Then there exist

X1AX= diag(1, , n)

82 / 85


Spectrum and spectral radius


127/129

The set of all eigenvalues of Ais the spectrum ofA We note it(A)The spectral radius(A)is defined as :

(A) = max(A)

||

83 / 85


Link between singular value and eigen value

The Eigen value of the symetric matrix ATAare the squares of thesingular values ofA

A t i t i h l i l


128/129

A symetric matrix has real eigen values.

A positive definite matrix have positive eigen value.

ATAis of course symmetric.ATAis positive definite, sincexTATAx= (Ax)T(Ax), which is the

square of the eulidian norm of(Ax).Therefore ATAhas positive real eigenvalue.Has any matrix,A has a singular value decomposition :A= Udiag(1, , n)VT And we have

ATA= Vdiag(1,

, n)U

TUdiag(1,

, n)V

T

=Vdiag(21, , 2n)VT

84 / 85



129/129

Hopefully we (quickly) covered most of the matrix algebra that will beneeded in the rest of the class.

85 / 85

ddis slides 1

Documents