krylov-subspace methods - i

Krylov-Subspace Methods - I

Lecture 6

Alessandra Nardi

Thanks to Prof. Jacob White, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy

Last lecture review

• Iterative Methods Overview– Stationary– Non Stationary

• QR factorization to solve Mx=b– Modified Gram-Schmidt Algorithm– QR Pivoting– Minimization View of QR

• Basic Minimization approach• Orthogonalized Search Directions• Pointer to Krylov Subspace Methods

4M

3M

2M

1M

12 13 14r r r

1Q

2Q

3Q

4Q

11r

22r 23 24r r

33r

44r

34r

Last lecture reminderQR Factorization – By picture

For i = 1 to N “For each Target Column”

For j = 1 to i-1 “For each Source Column left of target”

end

end

1

ii i

irp p

iii iMpr Mp

Normalize

i ix x v p

i ip e

Tjj

Tiir p M Mp

i ji i jp p pr Orthogonalize Search Direction

QR Factorization – Minimization ViewMinimization Algorithm

Iterative Methods

Solve Mx=b minimizing the residual r=b-Mx

Stationary: x(k+1) =Gx(k)+c• Jacobi

• Gauss-Seidel

• Successive Overrelaxation

Non Stationary: x(k+1) =x(k)+akpk

• CG (Conjugate Gradient) A symmetric and positive definite

• GCR (Generalized Conjugate Residual)

• GMRES, etc etc

Iterative Methods - CG

kkTk

kTkk

k

kT

k

kTk

k

kkkk

kkkk

drr

rrrd

Mdd

rr

Mdrr

dxx

)()(

)1()1()1(

1

)()(

)()1(

)()1(

)(

)(

)(

Convergence is related to:– Number of distinct eigenvalues– Ratio between max and min eigenvalue

Why ?How?

• General Subspace Minimization Algorithm– Review orthogonalization and projection formulas

• Generalized Conjugate Residual Algorithm– Krylov-subspace – Simplification in the symmetric case.– Convergence properties

• Eigenvalue and Eigenvector Review– Norms and Spectral Radius– Spectral Mapping Theorem

Outline

Approximately Solve Mx b

10 1Approximate as a weighted sum of , ,...,k

kx w w w

1 1 0

0

kk k

i ii

r b Mx r Mw

1

0

k

ki i

i

x w

Residual Minimizing idea: pick ' to minimizei s

2

21 1 1 0

22

kTk k k

i ii o

r r r r Mw

Arbitrary Subspace MethodsResidual Minimization

221 0

20 2

Minimizing is easy ifk

ki i

i

r r Mw

0 1 0 1, ,..., , ,...,k kspan p p p span w w w

0 1Create a set of vectors , ,..., such thatkp p p

0 or is orthogonal to T

i j i jMw Mw Mw Mw

and 0T

i jMp Mp

Use Gram-Schmidt on Mwi’s!

Arbitrary Subspace MethodsResidual Minimization

Arbitrary Subspace MethodsOrthogonalization

k

ii

iT

i

iT

kkk

TT

TTT

TT

TTT

TT

TTT

pMpMp

MpMwwp

MpMp

MpMwMpMp

MpMp

MpMw)(Mp)(Mpppwp

MpMp

MpMwMpMppwp

p

0

11

121212

00

02020211200222

00

01010100111

00

)()(

)()( :generalIn

0)()( and

0 and

0)()( and

1

0

For 1 to Tk

k ik k iT

i i i

Mw Mpi k p w p

Mp Mp

0

1

0

Tk

ikiT

i i i

r Mpx p

Mp Mp

Arbitrary Subspace Solution Algorithm

1. Given M, b and a set of search directions: {w0,…,wk}

2. Make wi’s MMT orthogonal and get new search directions: {p0,…,pk}

3. Minimize the residual:

For i = 0 to k

For j = 1 to i-1

end

end

1

ii iMp

p p Normalize

i ip w

Tj jii

Tip p M pMpp Orthogonalize Search Direction

01ii

T

ii rx x pMp Update Solution

Arbitrary Subspace Solution Algorithm

Krylov Subspace

• How about the initial set of search directions {w0,…,wk} ?

• A particular choice that is commonly used is: {w0,

…,wk} {b, Mb, M2b…}

m(A,v) span{v, Av, A2v, …, Am-1v} is called Krylov Subspace

1 0 0

0

kk i

i ki

x M r M r

1 0 1 0 0

0

kk i

i ki

r r M r I M M r

kth order polynomial

Krylov Subspace Methods

k

iii

k wx0

1

},...,,{},..., 121 bMMbbspanw,wspan{w k

k

Krylov Subspace MethodsSubspace Generation

0Note: for any 0

0 1 0 0 0 00span , = span ,r r r Mr r Mr

The set of residuals also can be used as a representation of the Krylov-Subspace

Generalized Conjugate Residual AlgorithmNice because the residuals generate next search directions

1

11

0

Tkkjk

k jTj

j j

Mr Mpp r p

Mp Mp

Tkk

k T

k k

r Mp

Mp Mp

1k kk kx x p

1k kk kr r Mp

Determine optimal stepsize in kth search direction

Update the solution (trying to

minimize residual) and the residual

Compute the new orthogonalized search direction (by using the most

recent residual)

Krylov-Subspace MethodsGeneralized Conjugate Residual Method

(k-th step)

1

11

0

Tkkjk

k jTj

j j

Mr Mpp r p

Mp Mp

Tkk

k T

k k

r Mp

Mp Mp

1k kk kx x p

1k kk kr r Mp

Vector inner products, O(n)Matrix-vector product, O(n) if sparse

Vector Adds, O(n)

O(k) inner products, total cost O(nk)

If M is sparse, as k (# of iters) approaches n,3total cost ( ) (2 ) .... ( ) ( )O n O n O kn O n

Better Converge Fast!


(Computational Complexity for k-th step)

1 1

1 11 1

0

T Tk kkj kk k

k j k kT Tj k kj j

Mr Mp Mr Mpp r p p r p

Mp MpMp Mp

If k (# of iters ) n, then symmetric, sparse, GCR is O(n2 )

1If then T k jM M r Mp j k

Better Converge Fast!

An Amazing fact that will not be derived

Orthogonalization in one step


(Symmetric Case – Conjugate Gradient Method)

Summary

• What is an iterative non stationary method: x(k+1) =x(k)+akpk

• How search to calculate:– Search directions (pk)– Step along search directions (ak)

• Krylov Subspace GCR • GCR is O(k2n)

– Better converge fast!

Now look at convergence properties of GCR

0 jIf for all in GCR, thenj k 0 0

0 1span , ,..., span , ,...,1) kkp p p r Mr Mr

th1 02) is the k ( ) , o rderkk kx M r

21

2polynomial which minimizes kr

1 1 0 03) ( )k kkr b Mx r M M r

0 01( )k kI M M r M r

01where is the order poly 1

th

k M r k 21

12minimizing subject to 0 = 1 k

kr

Krylov Methods Convergence AnalysisBasic properties

Krylov Methods Convergence AnalysisOptimality of GCR poly

• GCR optimality property (key property of the

algorithm): GCR picks the best (k+1)-th order

polynomial minimizing and subject to:

)(1 Mk2

2

1kr

1)0(1 k

GCR Optimality Property

k+1 polynomial such that 0 =1

ThereforeAny polynomial which satisfies the

constraints can be used to get an upper bound on

1

0

kr

r

1 0k+1 k+1( ) r where is any ordertk hkr M

Krylov Methods Convergence AnalysisOptimality of GCR poly

Eigenvalues and eigenvectors of a matrix M satisfy

eigenvector

eigenvalue

Or, is an eigenvalue of ifi M is singulariM I

= 0ii uM I

ii iu uM

is an eigenvector of ifiu M

Eigenvalues and eigenvectors reviewBasic definitions

Almost all NxN matrices have N linearly independent Eigenvectors

1 2 3 Nu u u u

M

1 1 2 2 3 3 N Nu u u u

The set of all eigenvalues of M is known as the Spectrum of M

Eigenvalues and eigenvectors reviewA symplifying assumption

Almost all NxN matrices have N linearly independent Eigenvectors

MU U1 0 0

0 0

0 0 N

1 1orM MU U U U Does NOT imply distinct eigenvalues, can equal i j Does NOT imply is nonsingular M

Eigenvalues and eigenvectors reviewA symplifying assumption

Re

Im

i

The spectral Radius of M is the radius of the smallest circle, centered at the origin, which encloses all of

M’s eigenvalues

Eigenvalues and eigenvectors reviewSpectral radius

L2 (Euclidean) norm :

n

iixx

1

2

2

L1 norm :

L norm :

n

iixx

11

ii

xx max

12x

Unit circle

Unit square

1

1 11x

1

x

Eigenvalues and eigenvectors reviewVector norms

Vector induced norm : Axx

AxA

xx 1maxmax

n

iij

jAA

11

max

Induced norm of A is the maximum “magnification” of by

= max abs column sum

n

jij

iAA

1

max = max abs row sum

2A = (largest eigenvalue of ATA)1/2

x A

Eigenvalues and eigenvectors reviewMatrix norms

Theorem: Any induced norm is a bound on the spectral radius

max1 ll

l

Mxx

M

Proof:First pick , 1i i l

x u u

i i i i i il l lMu u u

Eigenvalues and eigenvectors reviewInduced norms

Given a polynomial

0 1p

pf x a a x a x

Apply the polynomial to a matrix

0 1p

pf M a a M a M

Then

spectrum f M f spectrum M

Useful Eigenproperties Spectral Mapping Theorem

Krylov Methods Convergence AnalysisOverview

where is any (k+1)-th order polynomial

subject to:

may be used to get an upper bound on

)(~1 Mk

0

1

r

r k

1)0(~1 k

01

01

101

1 )(~)()( rMrMrrMr kkk

kk

)(~1 Mk

Matrix norm property GCR optimality property

• Review on eigenvalues and eigenvectors– Induced norms: relate matrix eigenvalues to the

matrix norms– Spectral mapping theorem: relate matrix eigenvalues

to matrix polynomials

• Now ready to relate the convergence properties of Krylov Subspace methods to eigenvalues of M

Krylov Methods Convergence AnalysisOverview

Summary

• Generalized Conjugate Residual Algorithm– Krylov-subspace – Simplification in the symmetric case– Convergence properties

• Eigenvalue and Eigenvector Review– Norms and Spectral Radius– Spectral Mapping Theorem

krylov-subspace methods - i

Documents