By: David McQuilling;Jesus Caban

Deng Li

Jan.,31,2006CS521

Numerical Linear Algebra

Solving Linear Equations

!"

#$%

&=!

"

#$%

&!"

#$%

&

21

8

94

22

v

u

xA b

2194

822

=+

=+

vu

vu

Gaussian Elimination

!"

#$%

&

2194

822

Start with the matrix representation of Ax = b by adding the solution vector bto the matrix A

Now perform Gaussian elimination by row reductions to transform the matrix Ab into an upper triangular matrix that contains the simplified equations.

Find the first multiplier by dividing the first entry in the second row by the firstentry in the first row. This will allow you to transform the first entry in thesecond row to a zero by a row reduction, thus reducing the system ofequations.

Gaussian Elimination (cont.)

2194

!"

#$%

&'!

"

#$%

&

550

822

2194

822

)822(2!

550

Multiply the first row by the multiplier (4 / 2 = 2) and then subtract that fromthe second row to obtain the new reduced equation.

A = LU

!"

#$%

&=!

"

#$%

&!"

#$%

&!"

#$%

&

21

8

50

22

12

01

v

u

xL cU

The matrix resulting from the Gaussian elimination steps is actually Uconcatenated with the new right hand side c.

L is obtained from the multipliers used in the Gaussian elimination steps.

Solving Ux=c by back substitution

!"

#$%

&=!

"

#$%

&!"

#$%

&

5

8

50

22

v

u

822 =+ vu

822 =+ vu

3=u

62 =u!"

#$%

&=!

"

#$%

&=

1

3

v

ux

U x c

55 =v

1=v

Solution to Ax=b

The solution vector to Ux=c is actually thesolution to the original problem Ax=b.

!"

#$%

&=!

"

#$%

&!"

#$%

&

21

8

94

22

v

u

!"

#$%

&=!

"

#$%

&!"

#$%

&

21

8

1

3

94

22

A x b

Gaussian algorithm to find LU

After this algorithm completes, the diagonal and upper triangle partof A contains U while the lower triangle contains L. This algorithmruns in O(n3) time.

Gram-Schmidt Factorization

Yields the factorization A=QR Starts with n linearly independent vectors in A Constructs n orthonormal vectors

A vector x is orthonormal if xTx = 1 and ||x|| = 1 Any vector is made orthonormal by making it a unit vector in the same

direction, i.e. the length is 1. Dividing a vector x by its length (||x||) creates a unit vector in the direction of x

!!!

"

#

$$$

%

&

=

2

1

2

a 3212|||| 222=++=a

!!!

"

#

$$$

%

&

=

3/2

3/1

3/2

||||/ aa

Composition of Q and R

Q is the matrix with n orthonormal column vectors, which wereobtained from the n linearly independent vectors of A.

Orthonormal columns implies that QTQ = I

R is obtained by multiplying QT on both sides of A = QR

This equation becomes QTA = QTQR = I R = R

R becomes an upper triangular matrix because later q’s are orthogonal toearlier a’s, i.e. their dot product is zero. This creates zeroes in the entriesbelow the main diagonal.

Obtaining Q from A

Start with the first column vector from A and use that as your firstvector q1 for Q (have to make it a unit vector before adding it to Q)

To obtain the second vector q2, subtract from the second vector inA, a2, its projection along the previous qi vectors. The projection of one vector onto another is defined as (xTy / xTx ) x

where the vector y is being projected onto the vector x.

Once all qi have been found this way, divide them by their lengthsso that they are unit vectors. Now those are the orthonormalvectors which make up the columns of Q

Obtaining R from QTA

Once Q has been found, simply multiply A by thetranspose of Q to find R

Overall, the Gram-Schimdt algorithm is O(mn2)for an mxn matrix A.

Eigenvalues and Eigenvectors

Eigenvectors are unique vectors such that whenmultiplied by a matrix A, they do not change theirdirection only their magnitude, i.e. Ax=λx

Eigenvalues are the scalar multiples of the eigenvectors,i.e. the λ’s

Solving for eigenvalues of A

Solving for the eigenvalues involves solving det(A–λI)=0where det(A) is the determinant of the matrix A

For a 2x2 matrix the determinant is a quadratic equation,for larger matrices the equation increases in complexityso lets solve a 2x2

Some interesting properties of eigenvalues are that theproduct of them is equal to the det(A) and the sum ofthem is equal to the trace of A which is the sum of theentries on the main diagonal of A

Solving det(A-λI)=0

!"

#$%

&=

21

23A !

"

#$%

&

'

'='

(

((

21

23IA

451*2)2)(3()det( 2+!=!!!=! """""IA

0452

=+! ""

0)1)(4( =!! ""

41=! 1

2=!

Solving for the eigenvectors

Once you have the eigenvalues, λ’s, you can solve forthe eigenvectors by solving (A-λI)x=0 for each λi

In our example we have two λ’s, so we need to solve fortwo eigenvectors.

Now solve (A-λI)x = 0 for λ=4

0)4( =! xIA

021

21

2

1=!

"

#$%

&!"

#$%

&

'

'

x

x

0221=+! xx

0221=! xx

!"

#$%

&='=2

12

21xxx

Solve (A-λI)x=0 for λ=1

0)( =! xIA

011

22

2

1=!

"

#$%

&!"

#$%

&

x

x

02221=+ xx

021=+ xx

!"

#$%

&

'=('=

1

1

21xxx

Positive definite matrices Positive definite:

all eigenvalues are positive All pivots are positive xTAx > 0 is positive except

at x=0

Applications Ellipse: find the axes of the

a tilted ellipse

122

=++ cybxyax

Cholesky Factorization•Cholesky factorization factors an N*N , symmetric, positive-definitematrix into the product of a lower triangular matrix and its transpose.

•A = LLT

Least Square•It often happens that Ax = b has no solution

•Too many equations (more equations than unknowns)

•When the length of e is as small as possible, x is a least square solution

•Example: Find the closest straight line to three points (0,6), (1,0), and (2,0)•Line: b = C + Dt•We are asking for two numbers C and D that satisfy three equations•No straight line goes through those points•Choose the one that minimize the error

Axbe !=

Matrix Multiplication

Matrix Multiplication Matrix multiplication is commonly used in

graph theory numerical algorithms computer graphics signal processing

Multiplication of large matrices requires a lot ofcomputation time as its complexity is O(n3), where n isthe dimension of the matrix

Most parallel matrix multiplication algorithms use matrixdecomposition based on the number of processorsavailable

Matrix Multiplication

!"

#$%

&=

!!!!

"

#

$$$$

%

&

!"

#$%

&

210184158

908070

121110

987

654

321

8765

4321

Example (a11):

( ) 7010*47*34*21*1

10

7

4

1

432111

=+++=

!!!!!

"

#

$$$$$

%

&

=a

Sequential Matrix Multiplication

The product C of the two matrices A and B isdefined by

kj

n

k

ikij bac !="#

=

1

0

aij, bij, and cij is the element in ith row and jth column ofthe matrix A, B, and C respectively.

Practice:

Make a example for A=LU(2x2), show meL and U.

Strassen’s Algorithm

The Strassen’s algorithm is a sequentialalgorithm reduced the complexity of matrixmultiplication algorithm to O(n2.81).

In this algorithm, n × n matrix is divided into 4n/2 × n/2 sub-matrices and multiplication is donerecursively by multiplying n/2 × n/2 matrices.

Parallel Matrix Partitioning

Parallel computers can be used to process largematrices.

In order to process a matrix in parallel, it is necessary topartition the matrix so that the different partitions can bemapped to different processors.

Partitions: Block row-wise/column-wise striping Cyclic row-wise/colum-wise striping Block checkerboard striping Cyclic checkerboard

Striped Partitioning

Matrix is divided into groups of complete rows orcolumns, and each processor is assigned one suchgroup.

Striped Partitioning Block-Striped: each processor is assigned

contiguous rows or columns Cyclic-Striped: is when rows or columns are

sequentially assigned to processors in awraparound manner.

Block Stripping(Application examples: (a) is in the case of car front crash,(b) is in the case of car side crash simulation.)

From: http://www.iitk.ac.in/cc/param/tpday21.html

Checkerboard Partition The matrix is divided into smaller square or

rectangular blocks or sub-matrices that aredistributed among processors

Checkerboard Partitioning Block-Checkerboard: partitioning splits both the

rows and the columns of the matrix, so no processoris assigned any complete row or column .

Cyclic-Checkerboard: when it wraparound.

Checkerboard Partitioning

From: http://www.iitk.ac.in/cc/param/tpday21.html

Parallel Matrix-Vector Product Algorithm:

For each processor I Broadcast X(i) Compute A(i)*x

A(i) refers to the n by n/p block row that processor i owns Algorithm uses the formula

Y(i) = Y(i) + A(i)*X = Y(i) + Σj A(i)*X(j)

Parallel Matrix Multiplication

Algorithms: systolic algorithm Cannon's algorithm Fox and Otto's algorithm PUMMA (Parallel Universal Matrix Multiplication) SUMMA (Scalable Universal Matrix Multiplication) DIMMA (Distribution Independent Matrix

Multiplication)

Systolic Communication

Communication designwhere the data exchangeand communicationoccurs between thenearest-neighbors.

From: http://www-unix.mcs.anl.gov

Matrix Decomposition

To implement the matrix multiplication, the A andB matrices are decomposed into several sub-matrices.

Four methods of matrix decomposition anddistribution: one-dimensional decomposition two-dimensional

square decomposition general decomposition scattered decomposition.

One-dimensional Decomposition

The matrix is horizontallydecomposed. The ith processorholds ith Asub and Bsub andcommunicates them to two neighborprocessors, i.e., to the (i-1)th and(i+1)th processors.

The first and last processorcommunicate with each other as in aring topology.

p0 p1 p2 p3 p5p4 p6 p7

Two-Dimensional Decomposition

Square decomposition: The two-dimensional square decompositionMatrix is decomposed into squareprocessor template.

General decomposition: GeneralTwo-Dimensional Decompositionallow having 2*3, 4*3, etc.configurations.

Two-Dimensional Decomposition

Scattered decomposition: thematrix is divided into several setsof blocks. Each set of blockscontains as many elements asthe number of processors, andevery element in a set of blocks isscattered according to the two-dimensional processor templates.

Applications

Computer Graphics In computer graphics all the transformation are

represented as matrices and a set of transformations asa matrix multiplication.

Transformations: Rotation Translation Scaling Shearing

X Rotation

© Pixar

2D/3D Mesh

•Laplacian matrix L(G) of the graph G

•If you take the eigenvalues andeigenvectors of L(G), you can getinteresting properties as vibration ofthe graph.

From: http://www.cc.gatech.edu/

Computer VisionIn computer vision we can calibrate a camera to determine the relationshipbetween what appears on the image (or retinal) plane and where it islocated in the 3D world by using and representing the parameters asmatrices and vectors.

Software for NumericalLinear Algebra

LAPACK

LAPACK: Routines for solving systems ofsimultaneous linear equations, least-squaressolutions of linear systems of equations,eigenvalue problems, and singular valueproblems.

Link:http://www.netlib.org/lapack/

LINPACK

LINPACK is a collection of Fortran subroutinesthat analyze and solve linear equations andlinear least-squares problems. The packagesolves linear systems whose matrices aregeneral, banded, symmetric indefinite,symmetric positive definite, triangular, and tri-diagonal square.

Link:http://www.netlib.org/linpack/

ATLAS ATLAS (Automatically Tuned Linear Algebra

Software) provides portable performance andhighly optimized Linear Algebra kernels forarbitrary cache-based architectures.

Link:https://sourceforge.net/projects/math-atlas/

MathWorks

MATLAB: provideengineers, scientists,mathematicians, andeducators with anenvironment for technicalcomputing applications

NAG

NAG Numerical Library:solve complex problems inareas such as research,engineering, life and earthsciences, financial analysisand data mining .

NMath Matrix

NMath Matrix is an advanced matrix manipulationlibrary. Full-featured structured sparse matrixclasses, including triangular, symmetric, Hermitian,banded, tridiagonal, symmetric banded, andHermitian banded.

Nastran

program participates in NASA-sponsored projectto develop a unified approach to computerizedstructural analysis. The program became knownas NASTRAN (NASA Structural AnalysisProgram).

NX Nastran, http://www.ugs.com/index.shtml MSC Nastran http://www.mscsoftware.com/