linear algebra and matlab tutorial
Post on 16-Apr-2017
1.816 Views
Preview:
TRANSCRIPT
Linear Algebra and MATLAB Tutorial
Jia-Bin HuangUniversity of Illinois, Urbana-Champaign
www.jiabinhuang.comjbhuang1@Illinois.edu
Today’s class• Introduction to MATLAB
• Linear algebra refresher
• Writing fast MATLAB code
Working with MATLAB
Learning by doing• Let’s work through an example!
• Download the example script here• Topics
• Basic types in Matlab• Operations on vectors and matrices• Control statements & vectorization• Saving/loading your work• Creating scripts or functions using m-files• Plotting• Working with images
Questions?
Linear algebra refresher• Linear algebra is the math tool du jour
• Compact notation• Convenient set operations• Used in all modern texts• Interfaces well with MATLAB, numpy, R, etc.
• We will use a lot of it!!
Slides credits: Paris Smaragdis and Karianne Bergen
Filtering, linear transformation-> extracting frequency sub-bands
Vector dot products-> measure patch similarities
Hybrid Image Image Quilting Gradient Domain Fusion
Solving linear systems-> solve for pixel values
Image-based Lighting
Solving linear systems -> solve for radiance map
Video Stitching and Processing
Solving linear systems -> solve for geometric transform
Scalars, Vectors, Matrices, Tensors
How will we see these?• Vector: an ordered collection of scalars (e.g., sounds)
• Matrix: a two-dimensional collection of scalars (e.g., images)
• Tensor: 3D signals (e.g., videos)
Element-wise operations• Addition/subtraction
• Multiplication (Hadamard product)
• No named operator for element-wise division• Just use Hadamard with inverted elements
c = a + b;c = a – b;
c = a.*b;
c = a./b;
Which division?Left Right
Array division C = ldivide(A, B);C = A.\B;
C = rdivide(A, B);C = A./B;
Matrix division C = mldivide(A, B);C = A/B;
C = mldivide(A, B);C = A\B;
EX: Array division
A = [1 2; 3, 4];B = [1,1; 1, 1];
C1 = A.\B;C2 = A./B;
C1 = 1.0000 0.5000 0.3333 0.2500
C2 = 1 2 3 4
EX: Matrix divisionX = A\B;-> the (least-square) solution to AX = BX = A/B;-> the (least-square) solution to XB = A
B/A = (A'\B')'
Transpose• Change rows to columns (and vice versa)
• Hermitian (conjugate transpose)• Notated as • Y = X’; % Hermitian transpose• Y = X.’; % Transpose (without conjugation)
Visualizing transposition• Mostly pointless for 1D signals
• Swap dimensions for 2D signals
Reshaping operators• The vec operator
• Unrolls elements column-wise• Useful for getting rid of matrices/tensors• B = A(:);
• The reshape operator• B = reshape(A, sz);• prod(sz) must be the same as numel(A)
EX: Reshape
A = 1:12; B = reshape(A,[3,4]); B = 1 4 7 10 2 5 8 11 3 6 9 12
Trace and diag• Matrix trace
• Sum of diagonal elements
• The diag operator EX: trace A = [1, 2; 3, 4]; t = trace(A); Q: What’s t?
EX: diag x = diag(A); D = diag(x); Q: What’s x and D?
Vector norm (how “big" a vector is)• Taxicab norm or Manhattan norm
• Euclidean norm
• Infinity norm
• P-norm
Unit circle (the set of all vectors of norm 1)
Which norm to use?• Say residual = (measured value) – (model estimate)
• : if you want to your measurement and estimate to match exactly for as many samples as possible
• : use this if large errors are bad, but small errors are ok.
• : use this if no error should exceed a prescribed value .
Vector-vector products• Inner product
• Shorthand for multiply and accumulate
• Outer product:
Matrix-vector product • Generalizing the dot product
• Linear combination of the columns of A
Matrix-matrix product C = ABDefinition: as inner products
As sum of outer productsAs a set of matrix-vector products• matrix and columns of
• rows of and matrix
Matrix products• Output rows == left matrix rows• Output columns == right matrix columns
Matrix multiplication properties• Associative:
• Distributive:
• NOT communitive:
Linear independence• Linear dependence: a set of vectors is linear dependent if there
exists a set of scalars with at least one such that
i.e., one of the vectors in the set can be written as a linear combination of one or more other vectors in the set.
• Linear independence: a set of vectors is linear independent if it is NOT linearly dependent.
Basis and dimension• Basis: a basis for a subspace is a linear independent set of vectors
that span • and are both bases that span • Not unique
• Dimension dim(): the number of linearly independent vectors in the basis for
Range and nullspace of a matrix • The range (column space, image) of a matrix
• Denoted by • The set of all linear combination of the columns of
• The nullspace (kernel) of a matrix • Denoted by • The set of vectors z such that
Rank of • Column rank of : dimension of
• Row rank of : dimension of
• Column rank == row rank
• Matrix is full rank if
System of linear equations• Ex: Find values that satisfy
Solution:• Step 1: write the system of linear equations as a matrix equation
• Step 2: Solve for
Mini-quiz 1• What’s the and for the following linear equations?
Mini-quiz 2• What’s the , and for the following linear equation?
Solving system of linear equations• Given and , find such that • Special case: a square matrix • The solution: • The matrix inverse exists and is unique if and only if
• is a squared matrix of full rank
• “Undoes” a matrix multiplication
• Y
Least square problems• What if ?
• No solution exists such that
• Least Squares problem: • Define residual • Find vector that minimizes
|
Least square problems• Decompose into components with
• and • .
• Since is in , the orthogonal complement of , the residual norm
which is minimized when and
Normal equations• The least squares solution occurs when , or equivalently
• Normal equations: • If has full column rank, then is invertible. • Thus has a unique solution
• x = A\b;• x = inv(A’*A)*A’*b; % same as backslash operator
Questions?
Writing Fast MATLAB Code
Using the Profiler• Helps uncover performance problems
• Timing functions: • tic, toc
• The following timings were measured on - CPU i5 1.7 GHz- 4 GB RAM
• http://www.mathworks.com/help/matlab/ref/profile.html
Pre-allocation Memory
3.3071 s
>> n = 1000;
2.1804 s2.5148 s
Reducing Memory Operations
>> x = 4;>> x(2) = 7;>> x(3) = 12;
>> x = zeros(3,1);>> x = 4;>> x(2) = 7;>> x(3) = 12;
Vectorization
2.1804 s 0.0157 s139x faster!
Using Vectorization• Appearance
• more like the mathematical expressions, easier to understand.
• Less Error Prone• Vectorized code is often shorter.• Fewer opportunities to introduce programming errors.
• Performance: • Often runs much faster than the corresponding code containing loops.
See http://www.mathworks.com/help/matlab/matlab_prog/vectorization.html
Binary Singleton Expansion Function• Make each column in A zero mean>> n1 = 5000;>> n2 = 10000;>> A = randn(n1, n2);
• See http://blogs.mathworks.com/loren/2008/08/04/comparing-repmat-and-bsxfun-performance/
0.2994 s 0.2251 s
Why bsxfun is faster than repmat?- bsxfun handles replication of the array
implicitly, thus avoid memory allocation- Bsxfun supports multi-thread
Loop, Vector and Boolean Indexing• Make odd entries in vector v zero• n = 1e6;
• See http://www.mathworks.com/help/matlab/learn_matlab/array-indexing.html• See Fast manipulation of multi-dimensional arrays in Matlab by Kevin Murphy
0.3772 s 0.0081 s 0.0130 s
Solving Linear Equation System
0.1620 s 0.0467 s
Dense and Sparse Matrices
• Dense: 16.1332 s• Sparse: 0.0040 s
More than 4000x faster!
Useful functions: sparse(), spdiags(), speye(), kron().0.6424 s 0.1157 s
Repeated solution of an equation system with the same matrix
3.0897 s 0.0739 s 41x faster!
Iterative Methods for Larger Problems• Iterative solvers in MATLAB:
• bicg, bicgstab, cgs, gmres, lsqr, minres, pcg, symmlq, qmr• [x,flag,relres,iter,resvec] = method(A,b,tol,maxit,M1,M2,x0)
• source: Writing Fast Matlab Code by Pascal Getreuer
Solving Ax = b when A is a Special Matrix• Circulant matrices
• Matrices corresponding to cyclic convolution Ax = conv(h, x) are diagonalized in the Fourier domain>> x = ifft( fft(b)./fft(h) );
• Triangular and banded• Efficiently solved by sparse LU factorization>> [L,U] = lu(sparse(A)); >> x = U\(L\b);
• Poisson problems• See http://www.cs.berkeley.edu/~demmel/cs267/lecture25/lecture25.html
In-place Computation>> x=randn(1000,1000,50);
0.1938 s 0.0560 s3.5x faster!
Inlining Simple Functions
1.1942 s 0.3065 s
functions are worth inlining: - conv, cross, fft2, fliplr, flipud, ifft, ifft2, ifftn, ind2sub, ismember, linspace, logspace, mean, median, meshgrid, poly, polyval, repmat, roots, rot90, setdiff, setxor, sortrows, std, sub2ind, union, unique, var
y = medfilt1(x,5); 0.2082 s
Using the Right Type of Data“Do not use a cannon to kill a mosquito.”
double image: 0.5295 s
uint8 image: 0.1676 s
Confucius
Matlab Organize its Arrays as Column-Major• Assign A to zero row-by-row or column-by-column>> n = 1e4;>> A = randn(n, n);
0.1041 s2.1740 s 21x faster!
Column-Major Memory Storage>> x = magic(3)x = 8 1 6 3 5 7 4 9 2
% Access one column>> y = x(:, 1);
% Access one row>> y = x(1, :);
Copy-on-Write (COW)>> n = 500;>> A = randn(n,n,n);
0.4794 s 0.0940 s
Clip values>> n = 2000;>> lowerBound = 0;>> upperBound = 1;>> A = randn(n,n);
0.0121 s0.1285 s 10x faster!
Moving Average Filter• Compute an N-sample moving average of x>> n = 1e7;>> N = 1000;>> x = randn(n,1);
3.2285 s 0.3847 s
Find the min/max of a matrix or N-d array >> n = 500;>> A = randn(n,n,n);
0.5465 s
0.1938 s
Acceleration using MEX (Matlab Executable)• Call your C, C++, or Fortran codes from the MATLAB
• Speed up specific subroutines
• See http://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html
MATLAB Coder• MATLAB Coder™ generates standalone C and C++ code from
MATLAB® code
• See video examples in http://www.mathworks.com/products/matlab-coder/videos.html
• See http://www.mathworks.com/products/matlab-coder/
DoubleClass• http://documents.epfl.ch/users/l/le/leuteneg/www/MATLABToolbox/
DoubleClass.html
parfor for parallel processing• Requirements
• Task independent• Order independent
See http://www.mathworks.com/products/parallel-computing/
Parallel Processing in Matlab• MatlabMPI• multicore• pMatlab: Parallel Matlab Toolbox• Parallel Computing Toolbox (Mathworks)• Distributed Computing Server (Mathworks)• MATLAB plug-in for CUDA (CUDA is a library that used an nVidia board)
• Source: http://www-h.eng.cam.ac.uk/help/tpl/programs/Matlab/faster_scripts.html
Resources for your final projects• Awesome computer vision by Jia-Bin Huang
• A curated list of computer vision resources
• VLFeat• features extraction and matching, segmentation, clustering
• Piotr's Computer Vision Matlab Toolbox• Filters, channels, detectors, image/video manipulation
• OpenCV (MexOpenCV by Kota Yamaguchi)• General purpose computer vision library
Resources• Linear Algebra
• Linear Algebra Review and Reference• Linear algebra refresher course• Quick Review of Matrix and Real Linear Algebra
• MATLAB Tutorial• Resource collection• MATLAB F&Q
• Writing Fast MATLAB code• Techniques for Improving Performance by Mathwork• Writing Fast Matlab Code by Pascal Getreuer• Guidelines for writing clean and fast code in MATLAB by Nico Schlömer
top related