numerical algs
DESCRIPTION
Numerical Algs. Inner products, Matrix Multiplication, LU-Decomposition (factorization) Simplex Method, Iterative methods ( sqrt ). Optimization. Many problems in computing are of the form: Find the values that minimize (or maximizes) some function subject to constraints. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/1.jpg)
Numerical Algs
Inner products, Matrix Multiplication,
LU-Decomposition (factorization)Simplex Method,
Iterative methods (sqrt)
![Page 2: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/2.jpg)
Optimization
• Many problems in computing are of the form:– Find the values
that minimize (or maximizes) some function
subject to constraints
![Page 3: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/3.jpg)
Example (unconstrained)• Find mean and standard deviation of a normal distribution
that maximizes the probability of samples
where the probability is
![Page 4: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/4.jpg)
Example (unconstrained)• Find a quadratic polynomial
where we have made measurements
so that the mean squared error in our measurements is minimized.
![Page 5: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/5.jpg)
Example
• (Help me out here)
![Page 6: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/6.jpg)
Gradient Descent
![Page 7: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/7.jpg)
Newton Iteration
![Page 8: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/8.jpg)
Matrices• This REALLY should be review for you….
• A matrix is a 2D array of numbers (real or complex) – The number of rows is usually called – The number of columns is usually called
• Matrices are represented by upper case letters
• Matrix elements two subscripts – is the element in row and column of matrix – Indices start at 1, not zero– Sometimes elements are written lowercase, sometime uppercase. (sloppy)
• Matrix multiplication is special
![Page 9: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/9.jpg)
Matrix + Matrix• Addition/Subtraction is done element by element
![Page 10: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/10.jpg)
Matrix * Scalar
• Done in parallel• Symmetric ()
![Page 11: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/11.jpg)
Matrix Transpose
• Use superscript to indicate transpose.• Exchange the order of subscripts (rows
become columns)
![Page 12: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/12.jpg)
Slicing• In this class I will use slicing to indicate a sub-array.• My notation is something like Matlab’s.
• Concatenation– I will use the | operator to concatenate matrices: , , or
![Page 13: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/13.jpg)
Vectors• A vector is a single-column matrix.• We write them with lowercase boldface letters.
• A row-vector is a single-row matrix. We always write them as to indicate they are the transpose of a column matrix.
![Page 14: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/14.jpg)
Inner Product• Defined for two vectors
Other notations for inner product:, ,
Also called the ‘dot’ product.
![Page 15: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/15.jpg)
Types of Matrix• Zero • Identity • Symmetric • Lower Triangular • Strict Lower Triangular • Upper & Strict Upper • Banded
![Page 16: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/16.jpg)
Accessing Matrix Elements
If we know the dimensions of the array ahead of time we can use a fixed-size 2D array.
But usually the dimensions are decided at runtime.I cannot pass this as a parameter to a function that operates on arrays with a variable size
![Page 17: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/17.jpg)
Writing functions for 2D arrays
• We need to do some extra work to be able to pass matrices arount
![Page 18: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/18.jpg)
Accessing Matrix Elements
• The layout is ‘contiguous’ in memory.• The is row major order. • The difference between items on the same row and different columns is
_______• The difference between items on the same column and different rows is
_______• The index of the item at row and column is
________
![Page 19: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/19.jpg)
Accessing Matrix Elements
• Other API’s use different layout (Fortran, OpenGL, MatLAB, …)• This is column major order. • The difference between items on the same row and different columns is
_______• The difference between items on the same column and different rows is
_______• The index of the item at row and column is
________• LaPACK is the most commonly used library for math.• It is implemented in Fortran, and called from c / c++.
[0 2 41 3 5 ]→[0,1 ,2 ,3 ,4 ,5 ]
![Page 20: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/20.jpg)
Representing a slice (MatLAB style)Let denote the numbers
So
Consider int A[] = Row 1 of has indices 0:1:2. Column 1 of A has indices 0:3:9.
![Page 21: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/21.jpg)
Basic idea of a slice
• Keep track of the start index (or pointer)
• Keep track of the number of values (or end index)
• Keep track of the difference between consecutive values (stepsize)
![Page 22: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/22.jpg)
C++ Slice
• std::slice– Start– Size (number of items)– Step (difference between consecutive items)
• std::slice(0, 3, 2) => 0, 2, 4Valarray<double> x;…..x[slice(0,3,2)] => x[0], x[2], x[4]
• I don’t use std::slice often (I write my own matrix class or use another library)
• The idea is very important
![Page 23: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/23.jpg)
Matrix Views & Submatrices• What if we want a submatrix that is not a single row or
column?
How can we represent ?• We need a 2D version of a slice • Keep track of…
– Linear index of in this case
– Offset between consecutive rows is , in this case 3– Offset between consecutive columns is – Keep track of the sizes (in this case )
![Page 24: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/24.jpg)
Matrix View• Imagine that library code & functions always operate on views
of some other underlying data. • A vector view might need the following:
– A pointer to the memory – Offset to the first item– Offset between consecutive items– The number of items in the vector.
• A matrix view might need the following:– Pointer to the memory– Offset to the first item– Offset between consecutive rows– Offset between consecutive items– Number of rows– Number of columns
In the STL these are called std::slice_array but do not seem to be used often. There are many custom implementations of this idea though.
In the STL these are called std::gslice_array but do not seem to be used often. There are many custom implementations of this idea though.
![Page 25: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/25.jpg)
Matrix MultiplicationIf then
Dimensions must agree:The number of columns in A must match the number of rows in B
![Page 26: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/26.jpg)
• Given what is the cost of MATRIX-MULTIPLY?
![Page 27: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/27.jpg)
Inner Products
![Page 28: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/28.jpg)
Vector Norms
![Page 29: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/29.jpg)
Outer Products
![Page 30: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/30.jpg)
Valarrays
![Page 31: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/31.jpg)
Matrix Transpose
![Page 32: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/32.jpg)
Inverse matrix (2 x 2)
![Page 33: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/33.jpg)
LU Decomposition
![Page 34: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/34.jpg)
![Page 35: Numerical Algs](https://reader036.vdocuments.us/reader036/viewer/2022062305/56814f89550346895dbd42fe/html5/thumbnails/35.jpg)