linear algebra - 國立交通大學資訊工程學系yi/courses/linearalgebra... · 2010-12-08 ·...
TRANSCRIPT
LA
Linear Algebra
Chih-Wei Yi
Dept. of Computer ScienceNational Chiao Tung University
December 8, 2010
LA
Orthogonality
Section 1 The Scalar Product in Rn
Section 1 The Scalar Product in Rn
LA
Orthogonality
Section 1 The Scalar Product in Rn
Length and Distance
In what follows, u = (u1, u2, � � � , un), v = (v1, v2, � � � , vn),w = (w1,w2, � � � ,wn) are vectors in Rn, and c , d 2 R are scalars.
De�nition
1 The length of v, denoted by kvk, is de�ned as
kvk =�v21 + v
22 + � � �+ v2n
� 12 .
It is also called as magnitude or norm.
2 If kvk = 1, then the vector v is called a unit vector.3 The Euclidean distance between two vectors u and v is
d (u, v) = ku� vk .
LA
Orthogonality
Section 1 The Scalar Product in Rn
Theorem
Assume v is a vector and c is a scalar. We have
1 kcvk = jc j kvk.2 If v is a nonzero vector, then 1
kvkv is a unit vector and has thesame direction as v.
LA
Orthogonality
Section 1 The Scalar Product in Rn
Dot (Inner) Products
De�nition
The dot (inner) product of u and v is denoted by u � v and givenby u � v = u1v1 + u2v2 + � � �+ unvn. If u and v are written ascolumn vectors, then u � v = uT v.
Theorem
The following properties are true.
1 u � v = v � u.2 u � (v+w) = u � v+ u �w.3 c (u � v) = (cu) � v.4 u � u = kuk2.5 u � u � 0. u � u = 0 i¤ u = 0.
LA
Orthogonality
Section 1 The Scalar Product in Rn
The Cauchy-Schwarz Inequality
Theorem
ju � vj � kuk kvk. In other words,����� n∑i=1 uivi����� �
n
∑i=1u2i
! 12
n
∑i=1v2i
! 12
.
De�nition
The angle between two nonzero vectors u and v is given by
cos θ =u � v
kuk kvk , 0 � θ � π.
Two vectors u and v are orthogonal if u � v = 0.
LA
Orthogonality
Section 1 The Scalar Product in Rn
Properties of Triangles
Theorem (The Triangle Inequality)
For any two vectors u and v, we have ku+ vk � kuk+ kvk. Notethat the equality stands if and only if two vectors have the samedirection.
u+v
vu
Theorem (The Pythagorean Theorem)
Two vectors u and v are orthogonal if and only if
ku+ vk2 = kuk2 + kvk2 .
LA
Orthogonality
Section 1 The Scalar Product in Rn
Projection
De�nition (Projection)
If u and v are vectors, we have v = v1 + v2 for some unique v1and v2 such that v2 � u = 0 and v1 = cu for some scalar c . Herev1 is called the vector projection of v onto u, and
v1 =u � vkuk2
u.
In addition, the scalar projection of v onto u is
α =u � vkuk .
LA
Orthogonality
Section 4 Inner Product Spaces
Section 4 Inner Product Spaces
LA
Orthogonality
Section 4 Inner Product Spaces
Inner Product Spaces
De�nition
Let u, v, and w be vectors in a vector space V, and c be anyscalar. An inner product on V is a function that associate a realnumber hu, vi with each pair of vectors u and v and satis�es thefollowing axioms.
1 hu, vi = hv,ui.2 hu, v+wi = hu, vi+ hu,wi.3 c hu, vi = hcu, vi.4 hu,ui � 0, and hu,ui = 0 i¤ u = 0.
LA
Orthogonality
Section 4 Inner Product Spaces
Examples of Inner Product Spaces
Example
The dot product of vectors in Rn.
Example
Let f , g 2 C [a, b]. Then, hf , gi =R ba f (x) g (x) dx is an inner
product on C [a, b].
Example
Let A,B 2M2�2. Then,hA,Bi = a11b11 + a12b12 + a21b21 + a22b22 is an inner product onM2�2.
LA
Orthogonality
Section 4 Inner Product Spaces
Properties of Inner Products
Theorem
Let u, v, and w be vectors in an inner product space V, and c beany scalar.
1 h0, vi = hv, 0i = 0.2 hu+ v,wi = hu,wi+ hv,wi.3 hu, cvi = c hu, vi.
LA
Orthogonality
Section 4 Inner Product Spaces
Cauchy-Schwarz inequality
Theorem
Let u and v be vectors in an inner product space V. We have
hu, vi � kuk kvk ,
and the equality stands if and only if u = λv for some scalar λ.
Proof.
Consider the norm of vector u� λv
0 � hu� λv,u� λvi = kuk2 � 2λ hu, vi+ λ2 kvk2 .
Let λ = hu,vikvk2 . Then, we have 0 � kuk
2 � hu,vi2
kvk2 . So,
hu, vi2 � (kuk kvk)2. In other words, hu, vi � kuk kvk. Inaddition, the equality stands if and only if u� λv = 0.
LA
Orthogonality
Section 4 Inner Product Spaces
Geometric Elements
De�nition
Let u and v be vectors in an inner product space V.
The norm of u is kuk = hu,ui12 .
The distance between u and v is d (u, v) = ku� vk.The angle between two nonzero vectors is given by
cos θ =hu, vikuk kvk , 0 � θ � π.
u and v are orthogonal if hu, vi = 0.If kuk = 1, the u is called a unit vector. If u 6= 0, 1
kuku is aunit vector in the direction of u.
LA
Orthogonality
Section 4 Inner Product Spaces
Example
Let A,B 2M2�2 as follows A =�1 20 �1
�and
B =�3 �12 1
�. Find
1 hA,Bi.2 kAk and kBk.3 d (A,B).4 1
kAkA.
5 The angle between A and B.
Problem
If u 6= 0, prove that 1kuku is a unit vector.
LA
Orthogonality
Section 5 Orthonormal Sets
Section 5 Orthonormal Sets
LA
Orthogonality
Section 5 Orthonormal Sets
Orthogonal and Orthonormal
De�nition (Orthogonal Sets and Orthonormal Sets)
A set S of vectors in an inner product space V is called orthogonalif every pair of vectors in S is orthogonal. In addition, if everyvector in the set is a unit vector, then S is called orthonormal.
LA
Orthogonality
Section 5 Orthonormal Sets
Example
S =n�
1p2, 1p
2, 0�,��p26 ,
p26 ,
2p23
�,� 23 ,�
23 ,13
�ois an
orthonormal set in R3.
Example
In P3, with the inner product hp,qi = a0b0 + a1b1 + a2b2 + a3b3,the standard basis B =
�1, x , x2, x3
is orthonormal.
Example
In C [0, 2π], with the inner product hf , gi =R 2π0 f (x) g (x) dx ,
the set S = f1, sin x , cos x , sin 2x , cos 2x , � � � , sin nx , cos nxg forany n 2 Z+ is orthogonal.
LA
Orthogonality
Section 5 Orthonormal Sets
Orthogonal Implies Linearly Independent
Theorem
If S = fv1, v2, � � � , vng is an orthogonal set of nonzero vectors inan inner product space V, then S is linearly independent.
Proof.
Assume c1v1 + c2v2 + � � �+ cnvn = 0. Then,
hc1v1 + c2v2 + � � �+ cnvn, v1i = h0, v1i = 0, and
hc1v1 + c2v2 + � � �+ cnvn, v1i =n
∑i=1ci hvi , v1i = c1 kv1k2 .
Therefore, c1 = 0. Similarly, we can prove c2 = 0, � � � , cn = 0.Thus, S is an linearly independent.
LA
Orthogonality
Section 5 Orthonormal Sets
Orthogonal Implies Linearly Independent
Corollary
If V is an inner product space of dimension n, then any orthogonalset of n nonzero vectors is a basis for V.
LA
Orthogonality
Section 5 Orthonormal Sets
Coordinates Relative to an Orthonormal Basis
Theorem
If B = fv1, v2, � � � , vng is an orthonormal basis for an innerproduct vector space V, then
v = hv, v1i v1 + hv, v2i v2 + � � �+ hv, vni vn,
i.e. the coordinate representation of a vector v w.r.t. B is
[v]B = (hv, v1i , hv, v2i , � � � , hv, vni)T .
LA
Orthogonality
Section 5 Orthonormal Sets
Proof.
Assume v = c1v1 + c2v2 + � � �+ cnvn. Then,
hv, v1i = hc1v1 + c2v2 + � � �+ cnvn, v1i= c1 hv1, v1i+ c2 hv2, v1i+ � � �+ cn hvn, v1i= c1 kv1k2 = c1.
Therefore, c1 = hv, v1i. Similarly, we can provec2 = hv, v2i , � � � , cn = hv, vni.
LA
Orthogonality
Section 5 Orthonormal Sets
Example
B =�v1 =
� 35 ,45 , 0�, v2 =
�� 45 ,35 , 0�, v3 = (0, 0, 1)
is an
orthonormal basis for R3. Find the coordinate of w = (5,�5, 2)w.r.t. B.
Solution
hw, v1i = (5,�5, 2) �� 35 ,45 , 0�= �1;
hw, v2i = (5,�5, 2) ��� 45 ,35 , 0�= �7;
hw, v3i = (5,�5, 2) � (0, 0, 1) = 2. Thus, w = �v1 � 7v2 + 2v3and [v]B = (�1,�7, 2)
T .
Problem (Quizzes)
What happens if B is orthogonal instead of orthonormal? Can youdevelop a formula for the projection?
LA
Orthogonality
Section 2 Orthogonal Subspaces
Section 2 Orthogonal Subspaces
LA
Orthogonality
Section 2 Orthogonal Subspaces
Direct Sum
De�nition
Let S1 and S2 be two subspaces of V. If each vector v 2 V can beuniquely represented as v = s1 + s2 where s1 2 S1 and s2 2 S2,then V is the direct sum of S1 and S2, and you can writeV = S1 � S2.
Example
B = f(1, 0, 1) , (1, 1, 0) , (1, 0, 0)g is a basis for R3. IfS1 = span f(1, 0, 1)g and S2 = span f(1, 1, 0) , (1, 0, 0)g, thenR3 = S1 � S2.
Problem
If V is an n dimension vector space and V = S1 � S2, prove thatdim (S1) + dim (S2) = n.
LA
Orthogonality
Section 2 Orthogonal Subspaces
Orthogonal Subspaces
De�nition
The subspace S1 and S2 of Rn are orthogonal if v1 � v2 = 0 for allv1 2 S1 and v2 2 S2.
Example
In R3, let S1 = span fe1, e2g and S2 = span fe3g. Then, S1 andS2 are orthogonal.
LA
Orthogonality
Section 2 Orthogonal Subspaces
Orthogonal Complement
De�nition
If S is a subspace of Rn, then the orthogonal complement of S isthe set S? = fu 2 Rn : u � v = 0 for all v 2 Sg.
Problem
If S is a subspace, prove that S? is a subspace.
Example
Find the orthogonal complement of the column space of the matrix
A =
26641 02 01 00 1
3775.
LA
Orthogonality
Section 2 Orthogonal Subspaces
Some Properties
Theorem
Let S be a subspace of Rn. Then,
1 Rn = S� S?.2 dim (S) + dim (S?) = n.3 (S?)? = S.
LA
Orthogonality
Section 2 Orthogonal Subspaces
Proof.
Assume fw1,w2, � � � ,wkg is an orthogonal basis of S. For anyv 2 Rn, let
v1 =hv,w1ihw1,w1i
w1 +hv,w2ihw2,w2i
w2 + � � �+hv,wk ihwk ,wk i
wk ,
v2 = v� v1.
We have v = v1 + v2, v1 2 S and v2 2 S?. So Rn is the directsum of S and S?. Then, dim (S) + dim (S?) = n follows. It canbe proved that fw1,w2, � � � ,wkg is also a basis of (S?)?.
LA
Orthogonality
Section 2 Orthogonal Subspaces
Project onto a Subspace
De�nition
Let S be a subspace of Rn. Since Rn = S� S?, for any v 2 Rn,we have v = v1 + v2 for some unique v1 2 S and v2 2 S?. Thevector v1 is called the project of v onto the subspace S, and isdenoted as v1 = projSv.
Theorem
If fv1, v2, � � � , vkg be an orthonormal basis for the subspace S ofRn, then projSv = hv, v1i v1 + hv, v2i v2 + . . .+ hv, vk i vk .
LA
Orthogonality
Section 2 Orthogonal Subspaces
Distance to a Subspace
Theorem
Let S be a subspace of Rn and v 2 Rn. Then, for any u 2 S andu 6=projSv, we have kv� projSvk � kv� uk.
Example
Let v = (1, 1, 3) and S = span f(0, 3, 1) , (2, 0, 0)g.1 Find the project of v onto S.2 Find the (shortest) distance between v and S.
LA
Orthogonality
Section 2 Orthogonal Subspaces
Another De�nition of Projection
De�nition (Projection)
S is a subspace of an inner product space V, and v is a vector inV. projSv, called the projection of v onto S, is a vector in S suchthat kv� projSvk is minimum.
If B = fw1,w2, � � � ,wkg is an orthogonal basis of S,
projSv =hv,w1ihw1,w1i
w1 +hv,w2ihw2,w2i
w2 + � � �+hv,wk ihwk ,wk i
wk .
If B = fu1,u2, � � � ,ukg is an orthonormal basis of S,
projSv = hv,u1i u1 + hv,u2i u2 + � � �+ hv,uk i uk .
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Section 6 The Gram-SchmidtOrthogonalization Process
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Theorem (Gram-Schmidt Orthonormalization Process)
Let B = fv1, v2, � � � , vng be a basis for an inner product vectorspace V. Let B 0 = fw1,w2, . . . ,wng, where wi is given by
w1 = v1,
w2 = v2 �hv2,w1ihw1,w1i
w1,
w3 = v3 ��hv3,w1ihw1,w1i
w1 +hv3,w2ihw2,w2i
w2
�,
...
wn = vn ��hvn,w1ihw1,w1i
w1 +hvn,w2ihw2,w2i
w2 + � � �+hvn,wn�1ihwn�1,wn�1i
wn�1
�.
Then, B 0 is an orthogonal basis for V.
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Gram-Schmidt Orthonormalization Process (Cont.)
Let ui = wikwi k for i = 1, 2, � � � , n. Then,
B" = fu1,u2, � � � ,ung is an orthonormal basis for V.For any k = 1, 2, � � � , n,span (v1, v2, � � � , vk ) = span (u1,u2, � � � ,uk ).For any i = 1, 2, � � � , n� 1,
projspan(v1,v2,��� ,vi�1)vi= projspan(w1,w2,��� ,wi�1)vi
=hvi ,w1ihw1,w1i
w1 +hvi ,w2ihw2,w2i
w2 + � � �+hvi ,wi�1ihwi�1,wi�1i
wi�1.
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Example
Let x1 = �1, x2 = 0, x3 = 1. Find an orthonormal basis for P2 ifthe inner product on P2 is de�ne by
hp,qi =3
∑i=1p (xi ) q (xi ) .
Solution
Consider the basis�v1 = 1, v2 = x , v3 = x2
.
w1 = 1; kw1k =p12 + 12 + 12 =
p3;u1 =
w1kw1k
=1p3.
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Solution ((Cont.))
w2 = x � hx , 1ih1, 1i1 = x ; kw2k =q(�1)2 + 02 + 12 =
p2;
u2 =w2kw2k
=1p2x .
and
w3 = x2 �x2, 1
�h1, 1i 1�
x2, x
�hx , xi x = x
2 � 23; kw3k =
r23;
u3 =w3kw3k
=
p62
�x2 � 2
3
�.
Then,n
1p3, 1p
2x ,p62
�x2 � 2
3
�ois an orthonormal basis.
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Revised Gram-Schmidt Orthonormalization Process
Theorem
Let B = fv1, v2, � � � , vng be a basis for an inner product vectorspace V. Let B 0 = fu1,u2, . . . ,ung is an orthonormal basis for V,where ui is given by
w1 = v1,u1 =w1kw1k
;
w2 = v2 � hv2,u1i u1,u2 =w2kw2k
;
...
wn = vn � (hvn,u1i u1 + hvn,u2i u2 + � � �+ hvn,un�1i un�1) ;un =
wnkwnk
.
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Gram-Schmidt QR Factorization (Improved)
Theorem
If A is an m� n matrix of rank n, then A can be factored into aproduct QR, where Q is an m� n matrix with orthonormal columnvectors and R is an upper triangle n� n matrix whose diagonalentries are all positive.
Proof.
Let fq1,q2, . . . ,qng be the orthonormal basis of Col (A) derivedfrom the Gram-Schmidt process.
rij = hqi , aj i = qTi aj for 1 � i , j � n.
We have�
a1 = r11q1ak = r1kq1 + r2kq2 + � � �+ rkkqk for k = 2, � � � , n
.
Let Q =�q1 q2 � � � qn
�and R = [rij ].
What is the rij in the Gram-Schmidt orthonormal process.
LA
Orthogonality
Section 6 The Gram-Schmidt Orthogonalization Process
Example
Compute the Gram-Schmidt QR factorization of the matrix
A =
26641 �2 �12 0 12 �4 24 0 0
3775 .
Solution26641 �2 �12 0 12 �4 24 0 0
3775 =266415 � 2
5 � 45
25
15
25
25 � 4
525
45
25 � 1
5
377524 5 �2 10 4 �10 0 2
35
LA
Orthogonality
Section 3 Least Squares Problems
Section 3 Least Squares Problems
LA
Orthogonality
Section 3 Least Squares Problems
Problem (Least Square Problems)
Let A be an m� n matrix and b be a vector in Rm .
1 If b is not in the column space of A, the system Ax = b iswith no solution.
2 Find x 2 Rn such that kAx� bk is minimal.
Solution
Let S denote the column space of A. The minimum of kAx� bkis given if Ax = projSb. Assume Ax
0 = projSb. Then,Ax0 � b = (projSb)� b is orthogonal to S. Then,
AT�Ax0 � b
�= 0 =) ATAx0 = AT b.
Therefore, the problem is reduced to solving the n� n linearsystem of equations ATAx0 = AT b, called the normal equation ofthe least square problem Ax = b.
LA
Orthogonality
Section 3 Least Squares Problems
Example
Let A =
24 0 23 01 0
35 and b =24 113
35. Find the orthogonal projectof b onto the column space of A.
Solution
ATA =
�0 3 12 0 0
� 24 0 23 01 0
35 = � 10 00 4
�,
AT b =
�0 3 12 0 0
� 24 113
35 = � 62
�.
LA
Orthogonality
Section 3 Least Squares Problems
Solution ((Cont.))
The normal equation is�10 00 4
� �x1x2
�=
�62
�with solution
x =�x1x2
�=
� 3512
�. So the projection is given by
Ax =
24 0 23 01 0
35 � 3512
�=
24 19535
35.
LA
Orthogonality
Section 3 Least Squares Problems
Problem
Find the least square regression line y = c0 + c1x for (1, 0), (2, 1),and (3, 3).
Solution
From (1, 0), we have c0 + c1 = 0. From (2, 1), we havec0 + 2c1 = 1. From (3, 3), we have c0 + 3c1 = 3.
Let A =
24 1 11 21 3
35, x = � c0c1
�, and b =
24 013
35. The problem is
equivalent to the LSP Ax = b.
LA
Orthogonality
Section 3 Least Squares Problems
Problem
Find the least square regression quadratic polynomialy = c0 + c1x + c2x2 for the data in the following table, and usethe model to estimate the value of y for x = 30.
x 0 5 10 15 20y 4.5 4.9 5.3 5.7 6.1
LA
Orthogonality
Section 3 Least Squares Problems
Solution
c0 = 4.5c0 + 5c1 + 25c2 = 4.9c0 + 10c1 + 100c2 = 5.3c0 + 15c1 + 225c2 = 5.7c0 + 20c1 + 400c2 = 6.1
=)
2666641 0 01 5 251 10 1001 15 2251 20 400
37777524 c0c1c2
35 =2666644.54.95.35.76.1
377775
LA
Orthogonality
Section 3 Least Squares Problems
Solution ((Cont.))
The normal equation ATAx = AT b are24 5 50 7550 750 12500750 12500 221250
3524 c0c1c2
35 =24 26.52854375
35
and the solution is x =
24 c0c1c2
35 �24 4.50.080
35. Therefore,y = 4.5+ 0.08x. If x = 30, we have y = 4.5+ 0.08� 30 = 6.9.
LA
Orthogonality
Section 7 Orthogonal Polynomials
Section 7 Orthogonal PolynomialsNOT COVERED IN THIS CLASS