review

ReviewPaul Heckbert

Computer Science DepartmentCarnegie Mellon University

15-859B - Introduction to Scientific Computing

y=y


y=-y, stable but slow solution


y=-y, unstable solution


Jacobian of ODEODE: y=f(t,y), where y is n-dimensional

Jacobian of f isa square matrix

if ODE homogeneous and linear then J is constant and y=Jy

but in general J varies with t and y


Stability of ODE depends on JacobianAt a given (t,y) find J(t,y) and its eigenvalues

find rmax = maxi { Re[i(J)] }

if rmax0, ODE unstable, locally


Stability of Numerical SolutionStability of numerical solution is related to, but not the same as stability of ODE!

Amplification factor of a numerical solution is the factor by which global error grows or shrinks each iteration.


Stability of Eulers MethodAmplification factor of Eulers method is I+hJNote that it depends on h and, in general, on t & y.Stability of Eulers method is determined by eigenvalues of I+hJspectral radius (I+hJ)= maxi | i(I+hJ) |if (I+hJ)

Implicit Methodsuse information from future time tk+1 to take a step from tkEuler method: yk+1 = yk+f(tk,yk)hkbackward Euler method: yk+1 = yk+f(tk+1,yk+1)hk

example:y=Ay f(t,y)=Ayyk+1 = yk+Ayk+1hk(I-hkA)yk+1=yk -- solve this system each iteration


Stability of Backward Eulers MethodAmplification factor of B.E.M. is (I-hJ)-1

B.E.M. is stable independent of h (unconditionally stable) as long as rmax

Large steps OK with Backward Eulers method


Popular IVP Solution MethodsEulers method, 1st orderbackward Eulers method, 1st ordertrapezoid method (a.k.a. 2nd order Adams-Moulton)4th order Runge-Kutta

If a method is pth order accurate then its global error is O(hp)


Solving Boundary Value ProblemsShooting Methodtransform BVP into IVP and root-finding

Finite Difference Methodtransform BVP into system of equations (linear?)

Finite Element Methodchoice of basis: linear, quadratic, ...collocation methodresidual is zero at selected pointsGalerkin methodresidual function is orthogonal to each basis function


2nd Order PDE ClassificationWe classify conic curve ax2+bxy+cy2+dx+ey+f=0 as ellipse/parabola/hyperbola according to sign of discriminant b2-4ac.Similarly we classify 2nd order PDE auxx+buxy+cuyy+dux+euy+fu+g=0:

b2-4ac < 0 elliptic(e.g. equilibrium)b2-4ac = 0 parabolic(e.g. diffusion)b2-4ac > 0 hyperbolic(e.g. wave motion)

For general PDEs, class can change from point to point


Example: Wave Equationutt = c uxx for 0x1, t0initial cond.: u(0,x)=sin(x)+x+2, ut(0,x)=4sin(2x)boundary cond.: u(t,0) = 2, u(t,1)=3c=1unknown: u(t,x)

simulated using Eulers method in tdiscretize unknown function:


Wave Equation: Numerical Solution

u0 = ...u1 = ...for t = 2*dt:dt:endtu2(2:n) = 2*u1(2:n)-u0(2:n) +c*(dt/dx)^2*(u1(3:n+1)-2*u1(2:n)+u1(1:n-1));u0 = u1;u1 = u2;end


Wave Equation Results


Poor results when dt too bigEulers method unstable when step too largedx=.05dt=.06


PDE Solution MethodsDiscretize in space, transform into system of IVPsDiscretize in space and time, finite difference method.Discretize in space and time, finite element method.Latter methods yield sparse systems.

Sometimes the geometry and boundary conditions are simple (e.g. rectangular grid);Sometimes theyre not (need mesh of triangles).


PDEs and Sparse SystemsA system of equations is sparse when there are few nonzero coefficients, e.g. O(n) nonzeros in an nxn matrix.

Partial Differential Equations generally yield sparse systems of equations.Integral Equations generally yield dense (non-sparse) systems of equations.

Sparse systems come from other sources besides PDEs.


Example: PDE Yielding Sparse SystemLaplaces Equation in 2-D: 2u = uxx +uyy = 0domain is unit square [0,1]2value of function u(x,y) specified on boundarysolve for u(x,y) in interior


Sparse Matrix StorageBrute force: store nxn array, O(n2) memorybut most of that is zeros wasted space (and time)!Better: use data structure that stores only the nonzeros col 1 2 3 4 5 6 7 8 9 10... val 0 1 0 0 1 -4 1 0 0 1... 16 bit integer indices: 2, 5, 6, 7,10 32 bit floats: 1, 1,-4, 1, 1

Memory requirements, if kn nonzeros:brute force: 4n2 bytes, sparse data struc: 6kn bytes


An Iterative Method: Gauss-SeidelSystem of equations Ax=bSolve ith equation for xi:Pseudocode:until x stops changingfor i = 1 to nx[i] (b[i]-sum{ji}{a[i,j]*x[j]})/a[i,i]modified x values are fed back in immediatelyconverges if A is symmetric positive definite


Conjugate Gradient MethodGenerally for symmetric positive definite, only.

Convert linear system Ax=binto optimization problem: minimize xTAx-xTba parabolic bowl

Like gradient descentbut search in conjugate directionsnot in gradient direction, to avoid zigzag problemBig help when bowl is elongated (cond(A) large)


Convergence ofConjugate Gradient MethodIf K = cond(A) = max(A)/ min(A)

then conjugate gradient method converges linearly with coefficient (sqrt(K)-1)/(sqrt(K)+1) worst case.

often does much better: without roundoff error, if A has m distinct eigenvalues, converges in m iterations!


Example: Poissons EquationSweep of Gauss-Seidel relaxes each grid value to be the average of its four neighbors plus an f offsetMany relaxations required to solve this on a fixed grid.Multigrid solves it on a hierarchy of grids.


Elements of Multigrid Methodrelax on a given grid a few times

coarsen (restrict) a grid

refine (interpolate) a grid


A Common Multigrid ScheduleFull Multigrid V Cycle:


Some Iterative MethodsGauss-Seidelconverges for all symmetric positive definite AConjugate Gradient (CG) Methodconvergence rate determined by condition numbernote that condition number typically larger for finer gridsPreconditioned Conjugate Gradientinstead of solving Av=f, solve M-1Av=M-1f where M-1 is cheap and M is close to Aoften much faster than CG, but conditioner M is problem-dependentMultigridconvergence rate is independent of condition number, problem sizebut algorithm must be tuned for a given problem; not as general as othersnote: dont need matrix A in memory can compute it on the fly!


Cost Comparisonon 2-D Poisson Equation, kk grid, n=k2 unknowns

METHOD COSTGaussian EliminationO(k6)= O(n3)Gauss-SeidelO(k4logk)= O(n2logn)Conjugate GradientO(k3)= O(n1.5)FFT/cyclic reductionO(k2logk) = O(nlogn)multigridO(k2) = O(n)optimal!


Function Representationssequence of samples (time domain)finite difference methodpyramid (hierarchical)polynomialsinusoids of various frequency (frequency domain)Fourier seriespiecewise polynomials (finite support)finite element method, splineswavelet (hierarchical, finite support)(time/frequency domain)


What Are Wavelets?In general, a family of representations using:hierarchical (nested) basis functionsfinite (compact) supportbasis functions often orthogonalfast transforms, often linear-time


Nested Function Spaces for Haar BasisLet Vj denote the space of all piecewise-constant functions represented over 2j equal sub-intervals of [0,1]

Vj has basis


Function Representations Desirable Propertiesgenerality approximate anything welldiscontinuities, nonperiodicity, ...adaptable to applicationaudio, pictures, flow field, terrain data, ...compact approximate function with few coefficientsfacilitates compression, storage, transmissionfast to compute withdifferential/integral operators are sparse in this basisConvert n-sample function to representation in O(nlogn) or O(n) time


Time-Frequency AnalysisFor many applications, you want to analyze a function in both time and frequencyAnalogous to a musical score

Fourier transforms give you frequency information, smearing time.Samples of a function give you temporal information, smearing frequency.

Note: substitute space for time for pictures.


Comparison to Fourier AnalysisFourier analysisBasis is globalSinusoids with frequencies in arithmetic progressionShort-time Fourier Transform (& Gabor filters)Basis is localSinusoid times GaussianFixed-width Gaussian windowWaveletBasis is localFrequencies in geometric progressionBasis has constant shape independent of scale


review

Documents

b introduction

scientific computingy

scientific computingexample

orderbackward eulers

ordertrapezoid method

tkeuler method

yk ftk

yk ayk