linear algebra and scientific pythonxiaowei/ai_materials... · linear algebra for machine learning....
TRANSCRIPT
![Page 1: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/1.jpg)
Linear Algebra and Scientific Python
Dr. Xiaowei Huang
https://cgi.csc.liv.ac.uk/~xiaowei/
![Page 2: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/2.jpg)
Topics
• Linear algebra• Scalars, vectors, matrices, tensors
• Multiplying matrices/vectors
• Introduction to Scientific Python • Numpy
• Scipy
• Matplotlib
![Page 3: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/3.jpg)
Linear Algebra For Machine Learning
![Page 4: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/4.jpg)
Scalar
• Single number
• Represented in lower-case italic x • E.g., let be the slope of the line
• Defining a real-valued scalar
• E.g., let be the number of units • Defining a natural number scalar
![Page 5: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/5.jpg)
Vector
• An array of numbers
• Arranged in order
• Each no. identified by an index
• Vectors are shown in lower-case bold
• If each element is in R then x is in Rn
• We think of vectors as points in space • Each element gives coordinate along an axis
![Page 6: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/6.jpg)
Matrix
• 2-D array of numbers
• Each element identified by two indices
• Denoted by bold typeface A
• Elements indicated as Am,n
• E.g.,
• A[i:] is ith row of A, A[:j] is jth column of A
• If A has shape of height m and width n with real-values then
![Page 7: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/7.jpg)
Tensor
• Sometimes need an array with more than two axes
• An array arranged on a regular grid with variable number of axes is referred to as a tensor
• Denote a tensor with bold typeface: A
• Element (i,j,k) of tensor denoted by Ai,j,k
![Page 8: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/8.jpg)
Transpose of a Matrix
• Mirror image across principal diagonal
• Vectors are matrices with a single column • Often written in-line using transpose
• Since a scalar is a matrix with one element
![Page 9: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/9.jpg)
Linear Transformation
• where and
n equations in n unknowns
![Page 10: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/10.jpg)
Linear Transformation
• where and • More explicitly
• Sometimes we wish to solve for the unknowns x ={x1,..,xn} when A and b provide constraints
Can view A as a linear transformation of vector x to vector b
![Page 11: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/11.jpg)
Identity and Inverse Matrices
• Matrix inversion is a powerful tool to analytically solve Ax=b
• Needs concept of Identity matrix
• Identity matrix does not change value of vector
• when we multiply the vector by identity matrix • Denote identity matrix that preserves n-dimensional vectors as In
• Formally In ∈ Rn×n and ∀x ∈Rn , In x = x
• Example of I3
![Page 12: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/12.jpg)
Matrix Inverse
• Inverse of square matrix A defined as A−1A=In
• We can now solve Ax=b as follows:
Ax=bA−1Ax = A−1b
In x = A−1b
x = A−1b
• This depends on being able to find A-1
• If A-1 exists there are several methods for finding it
![Page 13: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/13.jpg)
Solving Simultaneous equations
• Ax = b
• Two closed-form solutions • Matrix inversion x=A-1b
• Gaussian elimination
![Page 14: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/14.jpg)
Norms
• Used for measuring the size of a vector
• Norms map vectors to non-negative values
• Norm of vector x is distance from origin to x • It is any function f that satisfies:
![Page 15: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/15.jpg)
LP Norm
• Definition
![Page 16: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/16.jpg)
LP Norm
• Definition
• L2 Norm• Called Euclidean norm, written simply as
• Squared Euclidean norm is same as
![Page 17: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/17.jpg)
LP Norm
• Definition
• L1 Norm• also called Manhattan distance
![Page 18: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/18.jpg)
LP Norm
• Definition
• Norm• also called max norm
![Page 19: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/19.jpg)
Norms of two-dimensional Point
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0 0.5 1 1.5 2 2.5 3 3.5
Y-Values
||x||1 =
||x||2 =
||x||⚯ =
3+4 = 7
X = (3,4)
![Page 20: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/20.jpg)
Size of a Matrix
• Frobenius norm
• It is analogous to L2 norm of a vector
![Page 21: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/21.jpg)
Image distance
- =
- =
L1 distance between X and Y:
L2 distance between X and Y:
distance between X and Y:
![Page 22: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/22.jpg)
Introduction to Scientific Python
![Page 23: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/23.jpg)
Numpy
• Fundamental package for scientific computing with Python
• N-dimensional array object
• Linear algebra, Fourier transform, random number capabilities
• Building block for other packages (e.g. Scipy)
• Open source
![Page 24: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/24.jpg)
import numpy as np
• Basics:
• Slicing as usual
![Page 25: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/25.jpg)
More basics
numpy.linspace:evenly spaced numbers over a specified interval.
numpy.arange:evenly spaced values within a given interval.
![Page 26: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/26.jpg)
More basics
numpy.random.normal:Draw random samples from a normal (Gaussian) distribution
loc: meanscale: standard deviation
![Page 27: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/27.jpg)
Arrays are mutable
![Page 28: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/28.jpg)
Array attributes
numpy.reshape:Gives a new shape to an array without changing its data.
![Page 29: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/29.jpg)
Basic operations
• Arithmetic operators: elementwise application
• Also, we can use += and *=.
![Page 30: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/30.jpg)
Array broadcasting
• When operating on two arrays, numpy compares shapes. Two dimensions are compatible when• They are of equal size
• One of them is 1
![Page 31: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/31.jpg)
Array broadcasting
![Page 32: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/32.jpg)
Array broadcasting with scalars
• This also allows us to add a constant to a matrix or multiply a matrix by a constant
![Page 33: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/33.jpg)
Vector operations
• inner product
• outer product
• dot product (matrix multiplication)
![Page 34: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/34.jpg)
Matrix operations
• First, define some matrices:
![Page 35: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/35.jpg)
Matrix operations
np.dot(a,b)
If f both a and b are 1-D arrays, it is inner product of vectors
If both a and b are 2-D arrays, it is matrix multiplication
![Page 36: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/36.jpg)
Operations along axes
![Page 37: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/37.jpg)
Slicing arrays
• More advanced slicing
![Page 38: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/38.jpg)
Iterating over arrays
• Iterating over multidimensional arrays is done with respect to the first axis: for row in A
• Looping over all elements: for element in A.flat
![Page 39: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/39.jpg)
Reshaping
• Reshape • using reshape. Total size must remain the same.
• Resize • using resize, always works: chopping or appending zeros
• First dimension has ‘priority’, so beware of unexpected results
![Page 40: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/40.jpg)
Matrix operations
• import numpy.linalg
![Page 41: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/41.jpg)
Linear algebra
![Page 42: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/42.jpg)
Fourier transform
![Page 43: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/43.jpg)
Random sampling
![Page 44: Linear Algebra and Scientific Pythonxiaowei/ai_materials... · Linear Algebra For Machine Learning. Scalar •Single number •Represented in lower-case italic x •E.g., let be the](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1c74b03eaf3b5af72241c6/html5/thumbnails/44.jpg)
Distributions in random