LINEAR ALGBRA

Dan Witzner HansenEmail:witzner@itu.dk

LAST WEEK?

mnm

n

aa

aaa

A

1

11111

h d g c

f b e a

h g

f e

d c

b a

h d g c

f b e a

h g

f e

d c

b a

dh cf dg ce

bh af bg ae

h g

f e

d c

b a

MISC

Groups?

Improvements – what is missing?

TODAY

The goal is to be able to solve linear equations

Continue with linear algebra

Linear mappings

Basis vectors & independence

Solving linear equations & Determinants

Inverse & Least squares

SVD

Lot’s of stuff. Don’t despair – you will be greatly rewarded in the future

LINEAR SYSTEMS

WHAT IS A LINEAR EQUATION?

A linear equation is an equation of the form,

anxn+ an-1xn-1+ . . . + a1x1 = b.

WHAT IS A SYSTEM OF LINEAR EQUATIONS?

A system of linear equations is simply a set of linear equations. i.e.

a1,1x1+ a1,2x2+ . . . + a1,nxn = b1

a2,1x1+ a2,2x2+ . . . + a2,nxn = b2

. . .

am,1x1+ am,2x2+ . . . + am,nxn = bm

MATRIX FORM OF LINEAR SYSTEM

Compact notation A x = b

LINEAR MAPPINGS

A

Axy baxy

bAxy Affine mapping

EXAMPLE

Species 1: eats 5 units of A and 3 of B. Species 2: eats 2 units of A and 4 of B. Everyday a total of 900 units of A and

960 units of B are eaten. How many animals of each species are there?

€

5x1 + 2x2 = 900

3x1 + 4x2 = 960

Species

EXAMPLE:

€

5 2

3 4

⎡

⎣ ⎢

⎤

⎦ ⎥x1

x2

⎡

⎣ ⎢

⎤

⎦ ⎥=

900

960

⎡

⎣ ⎢

⎤

⎦ ⎥

Ax = b

€

5x1 + 2x2 = 900

3x1 + 4x2 = 960

MATLAB CODE

A = [5 2; 3 4];b = [900 960];

x = linspace(0,150,100);y1 = (-A(1,1)*x+b(1))/A(1,2); %made for clarityy2 = (-A(2,1)*x+b(2))/A(2,2);

Plot(x,y1,'r-','LineWidth',3); hold onPlot(x,y2,'b-','LineWidth',3); hold offtitle('Linear equations and their solution')

BASIS, INDEPENDENCE AND SUBSPACES

AN NOW FOR SOME FORMALISM

Subspaces Independent vectors Basis vectors / Orthonomal basis

SUBSPACES

A subspace is a vector space contained in another vector space

INDEPENDENT VECTORS

Axy Can it happen that y=0 if x is nonzero?

If y is non-zero for all non-zero x, then the column vectors of A are said to be linear independent.

These vectors form a set of basis vectors

Orthonormal basis when the vectors are unit size and orthogonal.

BASIS VECTORS - EXAMPLE

0

0

1

1e

0

1

0

2e

1

0

0

3e

4

3

2

1

0

0

4

0

1

0

3

0

0

1

2x

€

x =

2

3

4

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥=

1 0 0

0 1 0

0 0 1

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

2

3

4

⎡

⎣

⎢ ⎢ ⎢

⎤

⎦

⎥ ⎥ ⎥

Change of basis

SHOW THAT IT IS AN ORTHONORMAL BASIS

€

b1 =cosv

sinv

⎡

⎣ ⎢

⎤

⎦ ⎥,b2 =

−sinv

cosv

⎡

⎣ ⎢

⎤

⎦ ⎥

vv

vvB

cossin

sincos

WHAT HAPPENS WITH THIS ONE?

4123

682

241

A = [1 4 2;2 8 6; 3 12 4];[X,Y,Z] = meshgrid(-10:10,-10:10,-10:10);x = [X(:),Y(:),Z(:)]’;p = A*x;plot3(p(1,:),p(2,:),p(3,:),'rx')

A CLOSER LOOK AT MATRIX MULTIPLICATION

nmnmm

nn

nn

xaxaxa

xaxaxa

xaxaxa

Ax

2211

2222121

1212111

mn

n

n

n

mm a

a

a

x

a

a

a

x

a

a

a

x

2

1

2

22

12

2

1

21

11

1

nnAxAxAx :,2:,21:,1

2D EXAMPLE

€

Ax =a11 a12

a21 a22

⎡

⎣ ⎢

⎤

⎦ ⎥x1

x2

⎡

⎣ ⎢

⎤

⎦ ⎥

€

Ax =a11 a12

a21 a22

⎡

⎣ ⎢

⎤

⎦ ⎥x1

x2

⎡

⎣ ⎢

⎤

⎦ ⎥=x1

a11

a21

⎡

⎣ ⎢

⎤

⎦ ⎥+ x2

a12

a22

⎡

⎣ ⎢

⎤

⎦ ⎥

WHAT HAPPENS WITH THIS ONE?

4123

682

241

4x =

SOLVING LINEAR EQUATIONS

SOLUTIONS OF LINEAR EQUATIONS

A solution to a system of equations is simply an assignment of values to the variables that satisfies (is a solution to) all of the equations in the system.

If a system of equations has at least one solution, we say it is consistent.

If a system does not have any solutions we say that it is inconsistent.

RECALL

€

5 2

3 4

⎡

⎣ ⎢

⎤

⎦ ⎥x1

x2

⎡

⎣ ⎢

⎤

⎦ ⎥=

900

960

⎡

⎣ ⎢

⎤

⎦ ⎥

Ax = b€

5x1 + 2x2 = 900

3x1 + 4x2 = 960

Solution

SOLVING SYSTEMS ALGEBRAICALLY

2x 3y z 5

y z 1

z 3Which solution(s)?

Can we always do this?

How many solutions are there?

DETERMINANT

For A (2x2 matrix)

When det A ≠0 a unique solution exists (nonsingular) When det A =0 the matrix is singular (lines same

slope) and are therefore the columns are linear dependent Coincident (infinitely many solutions) Parallel (no solutions)

Determinant can be used when solving linear equations (Cramers’ rule), but not useful in practice

12212211det aaaaA >>det(A)

WHAT IF?

What to do when the dimension and the number of data points is large?

How many data points are needed to solve for the unknown parameters in x?

MATRIX INVERSE

Solve simple linear equation

Matrix inverse:

A (unique) inverse exist if det(A) ≠ 0 (NxN matrices)

Matlab: >>invA =inv(A)

€

ax = b

a−1ax = a−1bx

x = a−1b

0000

000

0010

0001

1

IAA

SOLVING LINEAR SYSTEMS

If m = n (A is a square matrix & Det(A)!=0), then we can obtain the solution by simple inversion (:

If m > n, then the system is over-constrained and A is not invertible

If n>m then under constrained.

MATRIX INVERSE EXAMPLE

Solve Ax=b (notice multiply from right):

6

4,

24

35

2

1,

54

32 1 bAA

bAx 1

NOTICE: IMPLEMENTATION

Don’t use for solving the linear system. It is mostly meant for notational convenience.

It is faster and more accurate (numerically) to write (solve) x=A\b than inv(A)*b:

bA 1

1111)( ABCABC

TT AA )()( 11

SIMPLE INVERSION OF (SOME) MATRICES Diagonal matrices

Orthogonal matrices

nnd

d

d

D

000

000

000

000

22

11

T

T

AA

IAA

1

,cossin

sincos,

cossin

sincos 1

TRRR

1

122

111

1

000

000

000

000

nnd

d

d

D

FITTING LINES

A 2-D point x = (x, y) is on a line with

slope m and intercept b if and only if y =

mx + b Equivalently,

So the line defined by two points x1, x2 is

the solution to the following system of equations:€

x 1[ ]m

b

⎡

⎣ ⎢

⎤

⎦ ⎥= y

€

x1 1

x2 1

⎡

⎣ ⎢

⎤

⎦ ⎥m

b

⎡

⎣ ⎢

⎤

⎦ ⎥=y1

y2

⎡

⎣ ⎢

⎤

⎦ ⎥

EXAMPLE: FITTING A LINE

Suppose we have points (2, 1), (5, 2), (7, 3), and (8, 3)?????

3

3

2

1

18

17

15

12

b

m

FITTING LINES

With more than two points, there is no guarantee that they will all be on the same line

courtesy ofVanderbilt U.

LEAST SQUARESFxy

)(,minargˆ Fxyrrx Objective:

Find the vector Fx in the column range of F, which is closest to the right-hand side vector y.The residual r=y-Fx

FITTING LINES

courtesy ofVanderbilt U.

Solution: Use the pseudoinverse

A+ = (ATA)-1AT to obtain least-

squares solution x = A+b

EXAMPLE: FITTING A LINE

Suppose we have points (2, 1), (5, 2), (7, 3), and (8, 3)

Then????

and x = A+b = (0.3571, 0.2857)T

EXAMPLE: FITTING A LINE(2, 1), (5, 2), (7, 3), and (8, 3)

HOMOGENEOUS SYSTEMS OF EQUATIONS

Suppose we want to solve A x = 0 There is a trivial solution x = 0, but we

don’t want this. For what other values of x is A x close to 0?

This is satisfied by computing the singular value decomposition (SVD) A = UDVT (a non-negative diagonal matrix between two orthogonal matrices) and taking x as the last column of V In Matlab [U, D, V] = svd(A)

SINGULAR VALUE DECOMPOSITION

Tnnnmmmnm VΣUA

IUU T

021 n

IVV T

nm

XXVT XVΣ T XVUΣ T

TTTnnn VUVUVUA 222111

000

00

00

00

Σ

2

1

n

NULL SPACE & IMAGE SPACE

4123

682

241

PROPERTIES OF SVD

When the columns of A =UDV are independent then all

Tells how close to singular A is. Inverse and pseudoinverse The columns of U corresponding to

nonzeros singular values span the range of A, the columns of V corresponding to zero singular values the nullspace.

€

dii > 0

0000

0000

000

000

Σ 2

1

TUVDA 11 TUVDA 1

0

EXAMPLE: LINE-FITTING AS A HOMOGENEOUS SYSTEM

A 2-D homogeneous point x = (x, y, 1)T is on the line l = (a, b, c)T only when

ax + by + c = 0

We can write this equation with a dot product:

x .l = 0, and hence the following system is implied for

multiple points x1, x2, ..., xn:

EXAMPLE: HOMOGENEOUS LINE-FITTING

Again we have 4 points, but now in homogeneous form:

(2, 1, 1), (5, 2, 1), (7, 3, 1), and (8, 3, 1)

The system of equations is:

Taking the SVD of A, we get:compare to x = (0.3571, 0.2857)T

ROBUST METHODS

So what about outliers

• Other metrics such as other norms• More about this later

NEXT WEEK