Lecture 15: Review of vectors and matricesDANIEL WELLER

THURSDAY, MARCH 7, 2019

AgendaLinear algebra and information

Vectors

Matrices and matrix operations

This diagram illustrates an important mathematical identity relating the lengths of vectors in a parallelogram. This allows us to easily compute the lengths of vectors added or subtracted together.

2

Image credit: Wikipedia/user:CheCheDaWaff

Introduction to linear algebraLinear algebra is the study of linear equations and operations and is foundational to many elements of mathematics, science, and engineering.

◦ In the abstract sense, linear algebra describes the theory of objects called vectors and operations involving these objects.

◦ In the practical sense, linear algebra concerns the computations involving such objects, including solving linear equations and analyzing vectors and matrices.

Some exciting areas that require a deep foundation in linear algebra:◦ Machine learning

◦ Data science

◦ Robotics

◦ Computer vision, and processing of images and signals

3

Nao robots know the importance of linear algebra.

Introduction to linear algebraWhat is a linear equation? A linear equation describes a line or plane in two or more dimensions:

4

2𝑥 − 3𝑦 = 1

𝑥 + 𝑦 − 𝑧 = 4

Introduction to linear algebraLinear algebra also concerns itself with linear operations.

Recall: properties of linearity (scaling, additivity). An operation on a vector that produces another vector, satisfying these properties, is linear.

◦ Examples we’ve already seen: Fourier spectrum, Fourier transform, linear filter, expected value

◦ Matrices describe another example of linear operation.

Thus, linear algebra defines three kinds of objects:◦ Vector: the elementary object involved in linear algebra; for instance, a pair of coordinates (x,y).

◦ Vector space: a collection of vectors that satisfies certain properties.

◦ Linear operator: a mapping from one vector space to another that satisfies linearity.

Abstract linear algebra will work on theoretical versions of such objects; we’ll work with concrete examples.

5

Linear algebra and informationLinear algebra connects to our study of information in multiple ways.

◦ System theory – we can analyze the frequency response and interpret the behavior of linear time-invariant filters. The frequency response can be viewed as eigenvalues of a matrix.

◦ Signal analysis – linear algebra can decompose or analyze signals by expressing them as a linear combination of simpler components. One such approach is called principal component analysis.

◦ Information preservation – we will see how we can use linear algebra to motivate a process of encoding and detecting or correcting errors in transmitted signals, preserving information that may otherwise have been lost.

After the break, we will build up towards this idea of error correction through linear algebra.

6

Linear algebra and informationHere is a sneak peak:

Consider a vector of 3 bits (a, b, c) received from a transmitter. Suppose there is some small chance that one of these bits were decoded wrong by thresholding. Can we determine if an error occurred?

Now, suppose the transmitter guarantees that a + b + c will add up to an even number if correct. What errors can we detect now?

More generally, by computing a series of linear equations, we can detect and even correct errors.

7

These error correcting codes were first developed for detecting errors in punched cards used in early computers.

Image credit: Wikipedia/Pete Birkinshaw

VectorsWhat is a vector?

◦ A vector describes an ordered sequence of numbers that has some sense of “magnitude” and “direction”.

◦ For instance the 2-D coordinates (3,-4) is a vector, that has a magnitude of 5 (Euclidean distance) and a direction pointing in the 4th quadrant:

Generally speaking, a vector is a mathematical object that defines two arithmetic operations:

◦ Addition/subtraction: v1 + v2 = v3

◦ Scaling (by a single number or scalar): av1 = v2

8

v

VectorsSome examples of vectors:

(1,0,0) and (0,0,1)

3-2i (complex number as a vector)

9

Operations on vectorsHow do we add vectors?

◦ Example: (-1,0,1) + (-2,2,0)

How do we scale vectors?◦ Example: 2(3,-2)

10

Operations on vectorsWhat does it mean for a vector to have “magnitude” or “direction”.

Magnitude is a measure of length or size. Generally speaking, small vectors have small magnitudes, and vice versa.

Direction describes the geometry or angle of the vector.

So in 2D Cartesian coordinates, the vector would correspond to the hypotenuse of a right triangle. The magnitude is the length of the hypotenuse, and the angle is the direction.

11

magnitude

direction

The zero vectorThe zero vector 0 is special such that adding it to any other vector v yields v back: v+0 = v.

◦ We assume that the zero vector is unique.

◦ Any scaling of the zero vector should give the zero vector back.

◦ In common cases: for the 2D coordinate vector, it is just (0,0); for complex numbers, it is 0+0i.

◦ We say the zero vector has magnitude (or length) zero, but it does not have a definite direction.

A vector space is a certain collection of vectors: among other requirements, it must contain the zero vector.

12

Special operations on vectorsAlthough not necessarily defined, we will be working with vectors that have two special operations:

◦ Norm of a vector: measures the magnitude of the vector. The Euclidean norm of a vector 𝑎, 𝑏, 𝑐 =𝑎2 + 𝑏2 + 𝑐2 is one common example.

◦ Example: compute norm of (2,-1)

◦ Inner product of a pair of vectors: measures how much of one vector points in the direction of another: the dot product 𝑎1, 𝑏1 , 𝑎2, 𝑏2 = 𝑎1𝑎2 + 𝑏1𝑏2 is a common example.◦ Example: compute inner product of (-1,3,1) and (-1,2,-1)

13

Triangles and parallelogramsGenerally speaking, the norm of a sum of vectors is governed bythe triangle inequality:

Another useful property is the parallelogram law illustrated by thediagram on the right, for the squares of the usual Euclidean norm:

Another identity useful for norms and inner products of vectors isthe Cauchy-Schwarz inequality:

14

Image credit: Wikipedia/user:CheCheDaWaff

𝑥 + 𝑦 ≤ 𝑥 + 𝑦

𝑥 + 𝑦 2 + 𝑥 − 𝑦 2 = 2 𝑥 2 + 2 𝑦 2

𝑥, 𝑦 ≤ 𝑥 𝑦

Note about vector spacesA vector space is a collection of vectors that includes, at a minimum, the zero vector. In addition, the vector space must be closed under linear operations. That is, when scaling a vector in the vector space, or adding two vectors together already in the vector space, the result must also be in the vector space.

Some common vector spaces:◦ Set of all 2D coordinate pairs (or 3D coordinate pairs)

◦ The set of complex numbers (representing real and imaginary parts as vector components)

We can also construct a smaller vector space from a set of vectors by considering the set of vectors formed by linear combinations of those vectors. We can call this a subspace.

15

Finite vectorsGenerally we will consider finite vectors, that have a finite dimension, or number of elements.

◦ Vector of 2D Cartesian coordinates is two-dimensional (x,y)

◦ Vector of 3D Cartesian coordinates is three-dimensional (x,y,z)

◦ Example: what is the dimension of (4,-3,1,0,1)?

For finite vectors, we also have:◦ Column vector: elements arranged in a single column; a D-dimensional column vector is a Dx1 matrix

◦ Row vector: elements arranged in a single row; a D-dimensional row vector is a 1xD matrix

◦ That brings us to our next object: the matrix

16

The matrixA matrix is a finite, ordered sequence of numbers arranged in a rectangle with some number of rows and columns.

Example:

Example:

17

3 −11 2

−1 17 40 3

Some matrices are a bit more elaborate than others…

Image credit: Isaiah van Hunan/Flickr

What does a matrix represent?Abstractly, a matrix is simply a rectangular collection of numbers (real or complex). In the context of linear algebra, however, we can use matrices to represent linear operations on finite-dimensional vectors.

The catch: the rows/columns must agree.◦ Multiplying a vector x and a matrix A:

◦ For Ax, # of columns of A = # of rows of x

◦ For xA, # of rows of A = # of columns of x

18

1 −13 −4−1 2

21

=1 2 + −1 13 2 + −4 1−1 2 + 2 1

=120

−1 1 0−1 10 32 −2

= 1 2

Matrix as a linear operatorShow that the matrix-vector multiplication Ax satisfies linearity:

◦ Scaling

◦ Additivity

Note: this also holds for the vector-matrix multiplication xA.

19

Matrix multiplicationWe have seen a couple examples of multiplying vectors and matrices. The same process can be used to multiply two matrices together.

◦ Example:

◦ Example:

◦ Note: # of columns of left matrix = # of rows of right matrix

20

2 −3−1 41 0

−1 0 11 2 2

=−5 −6 −45 8 7−1 0 1

−1 12 2

−2 3 02 0 1

=

Matrix operationsA couple notes about matrix and matrix/vector multiplication:

◦ Associative, but not commutative: (AB)C = A(BC), but (AB) is not the same as (BA)

◦ Distributive: (A+B)C = AC+BC

◦ For each multiplication, # of columns of left matrix or vector = # of rows of right matrix or vector

Other matrix operations:◦ We can add or subtract matrices of the same size, element-by-element

◦ Matrices have norms as well, but they are defined a bit differently and aren’t as useful to us.

21

The four spaces of a matrixSuppose we have a matrix A with M rows and N columns. Then, we can define four vector spaces:

◦ Row space: the vector space formed by any linear combination of the M vectors corresponding to the rows of the matrix A. This vector space contains vectors of dimension N.

◦ Column space: the vector space formed by any linear combination of the N vectors corresponding to the columns of matrix A. This vector space contains vectors of dimension M.

◦ Null space: the vector space containing all N-dimensional vectors x such that Ax = 0.

◦ Left null space: the vector space containing all M-dimensional vectors y such that yA = 0.

It turns out these spaces tell us a lot about the underlying linear operation performed by A. To understand why, we need to introduce a couple more definitions about vectors.

22

Linear independenceSuppose we have a set of vectors {v1, v2, …, vN}. This set of vectors is linearly dependent if there exists some nontrivial scalars a1, a2, …, aN (not all equal to zero) such that

If not, i.e., the only set of scalars is a1 = a2 = … = aN = 0, then we call this set linearly independent.

Now, suppose these vectors were of finite dimension N. Then, at most N such vectors can be linearly independent.

If we have a collection of linearly independent vectors, we say this set of vectors forms a basisfor the subspace (“spanned”) formed by linear combinations of these vectors.

23

𝑎1𝑣1 + 𝑎2𝑣2 +⋯+ 𝑎𝑁𝑣𝑁 = 𝟎

Linear independenceExample: is this set linearly independent?

24

−101

,2−11

, 111

Rank of a matrixRecall: The row and column spaces of a matrix are formed from linear combinations of the rows/columns of the matrix.

◦ The row rank of the matrix is the maximum number of linearly independent rows of the matrix. If all the rows are linearly independent, we say the matrix has full row rank.

◦ The column rank of the matrix is the maximum number of linearly independent columns of the matrix. If all the columns are linearly independent, we say the matrix has full column rank.

The row rank and the column rank of the matrix are necessarily equal. Thus, we usually just call these the rank of the matrix. If the matrix is both full column rank and full row rank, then we say it is also full rank.

25

Rank of a matrixIf a matrix has 3 rows and 2 columns, what is its maximum possible rank? How about a 2-row, 3-column matrix?

◦ If a matrix does not have full column rank, then the null space of the matrix is nontrivial (bigger than just the zero vector). The number of linearly independent vectors in this null space is the nullity of the matrix.

◦ If a matrix does not have full row rank, then the left null space of the matrix is nontrivial.

The column rank of the matrix + the nullity of the matrix = the # of columns. This is the rank-nullity theorem.

26

AnnouncementsDue today: Homework 5

Next week: Spring break

Next lecture: Matrix equations and more linear algebra stuff

27

Image credit: Patrick Verdier/Wikipedia