005background-linear algebra and probability

8/9/2019 005Background-Linear Algebra and Probability

1/46

Introduction to Linear Algebra


2/46

Matrix

representation

of size n X d


3/46

Symmetric Matrix

Diagonal Matrix


4/46

Matrix multiplication with a vector

gives a transformed vector


5/46

Inner Product

Norm of a

vector

Cosine of the angle

between 2 vectors

Cauchy Schwartz

Inequality


6/46

Linear Independence of vectors / Basis

vectors


7/46

Outer Product of 2 vectors


8/46

Rank of a matrix

Number of independent rows / columns of a

matrix

Outer product between 2 vectors gives a

matrix of rank 1.


9/46

Gradient of a scalar function


10/46

Gradient of a vector-valued

function (Jacobian)


11/46

Matrix / Vector Derivatives

Gradient of

Vector function

Gradient of scalar

function


12/46

Eigen values and vectors

Diagonalization property

1VDVM

V Matrix, whose columns correspond toEigenvectors of

D Diagonal matrix, corresponding toeigenvalues of

M

M


13/46

Rank deficient square matrices

A matrix Aof rank r will have r independentrows / columns, and is said to be rank-deficient.

There will be r independent eigenvectorscorresponding to r non- zero eigen

values

r eigen values are zero.


14/46

Matrix Inverse

For square matrix M


15/46

Orthogonal Matrix

IQQT

ji

jiqq

j

T

i 0

1

},.....,{ 21 Mqqq M vectors

Qcolumns of denote

Mqqq .....21

is of size M XM and is said to be

an orthogonal matrix

Q


16/46

Positive Definite Matrix

Positive definite square

matrix A0Axx

T

Eigen values of positive definite matrix A are

non-negative.

x denotes a vector of size d x 1 .

A denotes a vector of size d x d .


17/46

Real symmetric matrix

Elements of a matrix are real numbers

Matrix is symmetric

Eigen values of a real symmetric matrix arereal.


18/46

Probability Theory


19/46

Random variables

A random variable xdenotes a quantity that is

uncertain

May be result of experiment (flipping a coin) or a real

world measurements (measuring temperature) If observe several instances of xwe get different

values

Some values occur more than others and thisinformation is captured by a probability distribution

19


20/46

Discrete Random Variables

20


21/46


22/46


23/46

Continuous Random Variable

23


24/46

Expectation/ Variance of continuous

random variable


25/46

Joint Probability

Consider two random variables xand y

If we observe multiple paired instances, then some

combinations of outcomes are more likely than

others This is captured in the joint probability distribution

Written as P(x,y)

Can read P(x,y) as probability of xand y

25


26/46

For 2 random variables x and y,

26

Marginalization property


27/46

Marginalization

We can recover probability distribution of any variable in a joint distributionby integrating (or summing) over the other variables

27


28/46

Marginalization


28


29/46

Marginalization


Works in higher dimensions as wellleaves joint distribution betweenwhatever variables are left

29


30/46

Conditional Probability

Conditional probability of xgiven that y=y1is relative

propensity of variable xto take different outcomes given that

y is fixed to be equal to y1.

Written as Pr(x|y=y1)

30


31/46


Conditional probability can be extracted from joint probability Extract appropriate slice and normalize

31


32/46


More usually written in compact form

Can be re-arranged to give

32


33/46


This idea can be extended to more than two

variables

33


34/46

Bayes Rule

From before:

Combining:

Re-arranging:

34


35/46

Bayes Rule Terminology

Posteriorwhat we

know about yafter

seeingx

Priorwhat we knowabout ybefore seeing x

Likelihoodpropensity forobserving a certain value of

xgiven a certain value of y

Evidencea constant to

ensure that the left hand

side is a valid distribution35


36/46

Independence

If two variables xand yare independent then variable xtellsus nothing about variable y(and vice-versa)

36


37/46

Independence

When variables are independent, the joint factorizes into aproduct of the marginals:

37


38/46

Discrete Random vectors


39/46

Continuous Random vectors


40/46

Continuous random vectors


41/46

Discrete random vectors


42/46


43/46

Properties of covariance matrix


44/46

1 dimensional Gaussian


45/46

2 dimensional Gaussian


46/46

Multi dimensional Gaussian

005background-linear algebra and probability

Documents