chapter 5 orthogonality. outline scalar product in r n orthogonal subspaces least square problems...
Post on 19-Dec-2015
223 views
TRANSCRIPT
Chapter 5Orthogonality
Outline
Scalar Product in Rn
Orthogonal Subspaces Least Square Problems Inner Product Spaces Orthogonal Sets The Gram-Schmidt Orthogonalization Process
Scalar product in Rn
8
2
3
4
1
2
3
:Example
...2211
yx
yx
yxyxyxyx
T
nnT
Def: Let and be vectors in either R2 or R3.
The distance between and is defined to
be the number yx
x
y
x
y
Theorem 5.1.1
If and are two nonzero vectors in either R2 or
R3 and is the angle between them , then
cosTx xy y
x
y
Proof: By the law of cosines,
cos2222
yxyxxy
2 2 21cos
2x y x y y x
3 3 3
22 2
1 1 1
1
2 i i i ii i i
x y y x
x y3
1i i
i
x y
Corollary 5.1.2(Cauchy-Schwarz Inequality)
If and are vectors in either R2 or R3, then
With equality holding if and only if one of the
vectors is or one vector is a multiple of the
other.
Tx y x y
x
y
0
&x y
cosx y
x y
1 1
x y
x y
Note: If is the angle between , then
Thus
Def: The vectors and in R2(or R3)are said to
be orthogonal if .
6
4-
2
3 :B
, 0 :A
Examples2Rxx
x y
0Tx y
Scalar and Vector Projections
Scalar projection of onto :
Vector projection of onto :
1
T
T
T
x y
x y
y
x y
x yp u y y
y y y
x
u
y
z x p ������������� �
p u
cosx
Example: Find the point
on the line that
is closest to the point
(1,4)
Sol: Note that the vector is on the line
Thus the desired point is
xy3
1
3
1w
xy3
1
2.1
0.7
v ww
w w
v
Q
xy3
1
4.1
Example: Find the equation of the plane
passing through and
normal to
Sol:
4,3,2
3,1,2
0
3
1
2
4
3
2
z
y
x
0341322 zyx
Example: Find the distance form
to the plane
Sol: a normal vector to
the plane is
The distance
0,0,2P
022 zyx
2
2
1
3
2 P n
n
��������������
n
P��������������
Application 1: Information Retrieval Revisited Table 1
Frequency of Key words
Modules
Key Words M1 M2 M3 M4 M5 M6 M7 M8
determines 0 6 3 0 1 0 1 1
eignvalues 0 0 0 0 0 5 3 2
linear 5 4 4 5 4 0 3 3
matrices 6 5 3 3 4 4 3 2
numerical 0 0 0 0 3 0 4 3
orthogonality 0 0 0 0 4 6 0 2
spaces 0 0 5 2 3 3 0 1
systems 5 3 3 2 2 2 1 1
transformations 0 0 0 5 3 3 1 0
vector 0 4 4 3 2 2 0 3
Application I: Information Retrieval Revisited
A is the matrix corresponding to Table I, then the columns of the database matrix Q are determined by setting
To do a search for the key words orthogonality, spaces, vector, we form a unit search vector whose entries are all zero except for the three rows(be put in each of the rows) corresponding to the search rows.
x
1 1, ... ,8j j
j
q a ja
1
3
0.000 0.594 0.327 0.000 0.100 0.000 0.147 0.154
0.000 0.000 0.000 0.000 0.000 0.500 0.442 0.309
0.539 0.396 0.436 0.574 0.400 0.000 0.442 0.463
0.647 0.495 0.327 0.344 0.400 0.400 0.442 0.309
0.000 0.000 0.000 0.000 0.300 0.000 0.590 0.4Q
63
0.000 0.000 0.000 0.000 0.400 0.600 0.000 0.309
0.000 0.000 0.546 0.229 0.300 0.300 0.000 0.154
0.539 0.297 0.327 0.229 0.400 0.200 0.147 0.154
0.000 0.000 0.000 0.574 0.100 0.300 0.147 0.000
0.000 0.396 0.436 0.344 0.400 0.200 0.000 0.4
0.000
0.000
0.000
0.000
0.000
0.577
0.577
0.000
0.000
63 0.577
x
i
cos
where is the angle between the unit vectors and .
T Ti i i
i
y Q x y q x
x q
For our example,
0.000, 0.229, 0.567, 0.310, 0.635, 0.577, 0.000, 0.535T
y
Application I: Information Retrieval Revisited
Since is the entry of that is closest to 1,this indicates that the direction of the search vector is closest to the direction of and hence that Module 5 is the one that best matches our search criteria.
y
x
5q
5 0.635y
Application 2: Correlation And Covariance Matrices
Table 2
Math Scores Fall 1996
Scores
Student Assignment Exams Final
S1 198 200 196
S2 160 165 165
S3 158 158 133
S4 150 165 91
S5 175 182 151
S6 134 135 101
S7 152 136 80
Average 161 163 131
37 37 65
1 2 34
3 5 2
11 2 40
14 19 20
27 28 30
9 27 51
X
The column vectors of X represent the deviations from the mean for each of the three sets of scores.
The three sets of translated data specified by the column vectors of X all have mean 0 and all sum to 0.
A cosine value near 1 indicates that the two sets of scores are highly correlated.
Scale to make them unit vectors
Application 2: Correlation And Covariance Matrices
1 2 and x x
1 1 2 21 2
1 1 and u x u x
x x
If we set , thenTC U U
0.74 0.65 0.62
0.02 0.03 0.33
0.06 0.09 0.02
0.22 0.03 0.38
0.28 0.33 0.19
0.54 0.49 0.29
0.18 0.47 0.49
U
1 0.92 0.83
0.92 1 0.83
0.83 0.83 1
C
The matrix C is referred to as a correlation matrix.
The three sets of scores in our example are all positively correlated since the correlation coefficients are all positive.
A negative coefficient would indicate that two data sets were negatively correlated.
A coefficient of 0 would indicate that they were uncorrelated.
Application 2: Correlation And Covariance Matrices
Def: Two subspaces X and Y of are said to be orthogonal if = 0 for every and If X and Y are orthogonal, we write
x X y Y
X Y
5-2 Orthogonal Subspaces
31 2 3
Example:
Let { , } , { }
then
X span e e Y span e
X Y
Tx y
n
Def: Let Y be a subspace of . The set of all vectors in that are orthogonal to every vector in Y will be denoted . Thus = { for every } The set is called the orthogonal complement of Y
Y
| 0n Tx R x y Y y Y
Y
31
2 3
Example:
Let { }
then { , }
X span e
X span e e
nn
Remarks:
1. If X and Y are orthogonal subspaces of , then .
2. If Y is a subspace of , then is also a subspace of .
{0}X Y
Y
2Proof(1). If and , then 0
and hence 0.
Tx X Y X Y x x x
x
1 2
1 2 1 2
Proof(2). If and is a scalar, then for any ,
( ) ( ) 0 0
If and , then
( ) 0
T T
T TT
x Y y Y
x y x y x Y
x x Y
x x y x y x y
1 2
0 0 for each ,
.
Therefore, is a subspace of .n
y Y
x x Y
Y
nn n
Four Fundamental Subspaces
Let nmA
mnAor :x AxmnA
0n nN A x Ax
It will be shown later that N A R A , N A R A
and n N A R A
ARANm
for some m n mR A b b Ax x
0T m T mN A x A x
for some T n T m nR A b b A x x
Theorem 5.2.1(Fundamental Subspace Theorem)
pf: Let and
Also, if
Similarly,
x N A y R A
,: 0 1, ---------(1)A i x i m
1
and :, for some 's --------(2)m
i ii
y A i
(1)
1
:, 0m
ii
x y x A i
ARAN
N A R A
( ,:) 0, 1, 0 ( )A i z i m Az z N A
R A N A
hence, N A R A
:, 0, 1,z R A z A i i m
ARARAN
( ) (Let , then and ) ) ( .( )Tm n TN A R A N A R AA
Example: Let
Clearly,
00
21
02
01AA
1
0spanAN
1
2spanAN
2
1spanAR
0
1spanAR
ARAN
ARAN
Theorem 5.2.2
If S is a subspace of , then
Furthermore, if { } is a basis for S and
{ }is a basis for , then { , }
is a basis for .1,..., rx x
1,...,r nx x
S1,..., rx x
1,...,r nx x
n
n
dim dimS S n
Proof: If The result follows
Suppose . Let
and
{0} nS S
{0}S
1( ) n rrX x x
( ) ( )rank X r rank X ( )R X S
5.2.1
( ) ( )Theorem
S R X N X
3.6.4
dim( ) dim ( )Thm
S N X n r
To show that is a basis for ,
It remains to show their independency.
Let . Then
Similarly,
n
1
0n
i ii
c x
x S
1 1
0n r
i i i ii i
x c x x c x
1
0 0, 1r
i i ii
c x c i r
y S
1 1
0n n
i i i ii i r
y c x y c x
1
0 0, 1,n
i i ii r
c x c i r n
1{ }nx x
Def: If U and V are subspaces of a vector space W
and each can be written uniquely as a
sum , where and ,then we
say that W is a direct sum of U and V, and we
write
w W
u U
v V
W U V
u v
Theorem5.2.3: If is a subspace of ,
then
pf: By Theorem5.2.2,
To show uniqueness,
Suppose
where
nS SSn
SSn
, , & .nx x u v u S v S
1 1 2 2x u v u v
1 2 1 2, & ,u u S v v S
1 2 2 1u u v v S S {0}S S
1 2 2 1&u u v v
Theorem5.2.4: If is a subspace of ,
then
pf: Let
If
S nSS )(
rS )dim(
rSTheorem
)dim(2.2.5
, then 0x S x y y S
x S
SS
SS
Remark: Let . i.e. , Since
and
are bijections .
nmA mnA : ARANn
ArankArank )( ArankAnullityn ArankAnullity
ARARA :
ARARA :
Let nmA nA :
mA :
m
n
A
A
A
A
AN
AN
AN
AN
bijection
bijection
0
0
Cor5.2.5:
Let and . Then
either
(i)
or (ii)
pf:
nmA mb
nx Ax b
my
0 and 0A y y b
(i) ( ) nb R A x Ax b
(ii) ( ) ( ) ( ) 0b R A N A y N A y b
0 & 0my A y y b
( ) ( )m TR A N A )(AR
b
)( AN
for 3m
Example: Let . Find
The basic idea is that the row space and the sol. of
are invariant under row operations.
Sol: (i)
(Why?)
(ii)
(Why?)
(iii) Similarly,
and
(iv) Clearly,
431
110
211
A )(),(),(),( ARANARAN
Ax b
000
110
101
~ r
row
AA
1
1
0
1
0
1
)( spanAR
1 3 2 30 0 & 0rA x x x x x
1
1
1
)( spanAN
000
210
101
~row
A
2
1
0
1
0
1
)( spanAR
1
2
1
)( spanAN
)()(& ARANARAN
Example: Let
(i)
and
(ii) The mapping
and
(iv) What is the matrix representation for ?
23:0
0
3
0
0
2
A
0
1
0
0
0
1
1
0
03 spanspanARAN
2)( AR
( ): ( ) is a bijection
R AA R A R A
0
3
2
02
1
2
1
x
x
x
x
)(:1
)(
ARARAAR
03
12
1
2
1
2
1 y
y
y
y
)( ARA
5-4 Inner Product Spaces
A tool to measure the
orthogonality of two vectors in
general vector space
Def: An inner product on a
vector space is a function
Satisfying the following conditions:
(i) with equality iff
(ii)
(iii)
V
, : ( )V V F orC
, 0x x 0x
, ,x y y x
, , ,x y z x z y z
Example: (i) Let
Then is an inner product of
(ii) Let , Then is an
inner product of (iii) Let and then
is an inner product of
(iv) Let , is a positive function and are distinct real numbers. Then
is an inner product of
, & 0 1, .nix y w i n
1
,n
i i ii
x y w x y
n
ij
n
jij
m
i
baBA
11
,nmBA ,nm ].,[)(,, 0 baCxwgf 0)( xw
b
a
dxxgxfxwgf )()()(,
].,[0 baC
, np g P )(xw
nxx 1
)()()(,1
ii
n
ii xgxPxwgp
nP
Def: Let be an inner product of a
vector space and .
we say
The length or norm of is given
by
,
V ,u v V
, 0u v u v
v
,v v v
Theorem5.4.1: (The Pythagorean Law)
pf:
2 2 2u v u v u v
2 2u v
2,u v u v u v
, , , ,u u u v v u v v
u
v u v
Example 1: Consider with inner product
(i)
(ii)
(iii)
(iv) (Pythagorean Law)
or
]1,1[0 C
1
1)()(, dxxgxfgf
xxdxx 101,11
1
212111,11
1 dx
32
3
2,
1
1 xxdxxxx
3
8
3
2211
222 xx
3
8)1(1,11
21
1
2 dxxxxx
Example 2: Consider with inner product
It can be shown that
(i)
(ii)
(iii)
Thus is an orthonormal
set.
0[ , ]C
dxxgxfgf )()(
1,
0sin,cos mxnx
cos ,cos
sin ,sin
mn
mn
mx nx
mx nx
1,cos ,sin
2nx nx n N
1 1, 1
2 2
Remark
Remark: The inner product in example 2 plays a key
role in Fourier analysis application involving trigo-
nometric approximation of functions.
2 2cos sin cos sin 2x x x x
Example 3: Let
and let
Then is not orthogonal to
, ,m nA B
4
0
1
3
3
1
,
3
2
1
3
1
1
BA
ABA 6, B
6,5 FF
BA
1 1 1
, ( ) ( )
,
m n mT T
ij ij iii j i
F
A B a b trace AB AB
A A A
Def: Let be two vectors in an
inner product space . Then
the scalar projection of onto is
defined as
The vector projection of onto is
& 0u v
V
u
v
,1,
u vu v
vv
,1
,
u vp v v
v v v
u
v
Lemma: Let be the vector projection
of onto . Then
for some
pf:
0 &v p
u v
( )
( )
i u p p
i u p u kv
2 2
( ) , , ,
, , 0
, ,
( ) .
i p u p p u p p
u v u v
v v v v
p u p
ii trivial
u
v p
u p
k
Theorem5.4.2: (Cauchy-Schwarz Inequality)
Let be two vectors in an
inner product space . Then
Moreover, equality holds are linear dependent.
pf: If
If
Equality holds
i.e., equality holds iff are linear dependent.
&u v
V,u v u v
&u v
0, then , 0
0, then
v u v u v
v
2
2 2 2,
,
Pythagorean Theoremu vp u u p
v v
2 2 2 2 2 22,u v u v v u p u v
,0, or
,
u vv u p v
v v
&u v
u
v p
u p
Note: From Cauchy-Schwarz Inequality for .
This, we can define as the angle between the two nonzero vectors
,1 1 if 0 and 0
,! 0, cos .
u vu v
u v
u v
u v
& .u v
F
Def: Let be a vector space a function
is said to be a norm if it satisfies
V
: {0}V v v
( ) 0 with equality 0
( ) , scalar .
( ) (triangle inequality)
i v v
ii v v
iii v w v w
Remark: Such a vector space is called a normed linear space.
Theorem5.4.3: If is an inner product
space, then
defines a norm on
pf: trivial
Def: The distance between is defined
as.u v
V
.V
&u v
, v v v v V
Example: Let , thennx
11
1
1
1
2
21
( ) is a norm.
( ) is a norm.
( ) is a norm for any 1.
in particular =2,
, is the euclidean norm.
max
n
ii
ii n
n PP
iPi
n
ii
i x x
ii x x
iii x x p
p
x x x x
Example: Let
Thus,
However,
(Why?)
1 2
1 4&
2 2x x
1 2, 0x x
2 2 2
2 1 22 2 25 20 25x x x x
2 2
1 2
2
1 2
4 16
20 16
x x
x x
Remark: In the case of a norm that is not derived from an inner product, the Pythagorean Law will not hold.
Example: Let ,
then
4
5
3
x
1
2
12
5 2
5
x
x
x
Example: Let
Then
2 1B x x
1B2B B
1 1 1
5-3 Least Squares Problems
Least squares problems
A typical example: Given
Find the best line to fit the data . or
or find such that is minimum Geometrical meaning :
, 1,i
i
xi n
y
0 1y c c x 1 1
2 20
1
1
1
1 n n
x y
x ycsolve
c
x y
Ac y
0 1,c cAc y
0 1y c c x
( , )n nx y
1 1( , )x y
Least squares problems:
Given
then the equation
may not have solutions
The objective of least square problem is
trying to find such that
is minimum value
i.e., find satisfying
& ,m n mA b
Ax b
. ., ( ) ( )i e b Col A R A
x
b Ax
x
minnx
b A x b Ax
( )R A
b
Ax
Preview of the results:
It will be shown that
If columns of are linear independent .
( )! ( ), min
y R Ap R A b p y b
( ) ( )
0
0
b p R A N A
A b p
A b Ax
A Ax A b
1x A A A b
A
b
( )R Ap
Theorem5.3.1: Let be a subspace of , then
(i) for all
(ii)
pf:
(i)
where
If
(ii) follows directly from (i) by noting that
S m, ! , mb p S b y b p
\{ }y S p
miny S
p b y b b p S
m S S
b p z
&p S z S
\{ }y S p
2
2 2 2
0
Pythogorean Theorem
Sz S
b y b p p y b p p y
b p z S
p
b
S
unique expression
Question: How to find which solves
Ans.:
From previous Theorem , we know that
x
min ?nx
A x b b Ax
( ) ( )
( ) 0
0
b p R A N A
A b p
A b A Ax
( )R A
p Ax
b
Definition : is called normal equatio .ntheA Ax A b
Remark: In general, it is possible to have more than one solution to the normal equation. If is a solution, then the general solution is of the form
ˆ where ( )x h h N A
x̂
Theorem5.3.2: Let and
Then the normal equation
has an unique solution .
and is the unique least squares solution to
pf: To show that is nonsingular
nmA .)( nArank A Ax A b
1x A A A b
x
.Ax b
AA
Let 0 ( ) ( ) {0}
0 0 ( ( ) )
T TA Ax Ax N A R A
Ax x rank A n
1 is the unique solution.x A A A b
Note: The projection vector
is the element of that
is closet to in the least squares
sense .
Thus, The matrix is called the
projection matrix (that project any vector of
to )
1p Ax A A A A b
)(AR
b
1P A A A A
)(ARm
b
)(AR
p
Suppose a spring obeys the Hook’s law
and a series of data are taken (with measurement
error) as
How to determine ?
sol: Note that is inconsistent
The normal equation is
so,
KxF
11
8
7
5
4
3
x
F
8
5
3
11
7
4
811
57
34
Kor
K
K
K
8
5
3
1174
11
7
4
1174 K
K
186 135 0.726K K
Application 2: Spring Constants
Example 2: Given the data
Find the best least squares fit by a linear function.
sol: Let the desired linear function be The problem becomes to find the least squares solution of
is the unique solution.
Thus, the best linear least square fit is
5
6
4
3
1
0
y
x
xccy 10
0
1
1 0 1
1 3 4
1 6 5c
yA
c
c
10
1
4
32
3
cA A A y
c
xy3
2
3
4
∵ rank(A)=2
Example3: Find the best quadratic least squares fit to the data
sol:
Let the desired quadratic function be
The problem becomes to find the least square
solution of
is the unique solution.
Thus, the best quadratic least square fit is
0
1
1
2
2.75
0.25
0.25
c
c A A A y
c
4
3
4
2
2
1
3
0
y
x
2210 xcxccy
2
1
0
9
4
1
0
3
2
1
0
1
1
1
1
4
4
2
3
c
c
c
225.025.075.2 xxy
∵ rank(A)=3
5-5 Orthonormal Sets
Orthonormal Set
Simplify the least squares solution
(avoid computing inverse) Numerical computational stability
Def: is said to be an orthogonal set in
an inner product space if
Moreover, if , then is said
to be orthonormal.
1 nv v
V
, 0i jv v for all i j
,i j ijv v 1 nv v
Example 2:
is an
orthogonal set but not orthonormal.
However ,
is orthonormal.
1 2 3
1 2 4
1 , 1 , 5
1 3 1
v v v
1 1 2 2 3 3
1 1 1, ,
3 14 42u v u v u v
Theorem5.5.1: Let be an orthogonal
set of nonzero vectors in an inner product
space . Then they are linear independent.
pf: Suppose that
1 nv v
V
1
0n
i ii
c v
1 1
1
0 , , ,
0, 1,
is linearly independent.
n n
j i i i j i j j ji i
j
n
v c v c v v c v v
c j n
v v
Example:
is an
orthonormal set of with inner
product .
Note: Now you know the meaning what one
says that .
1, cos , sin
2nx nx n N
,0 C
dxxgxfgf )()(
1,
xx sincos
Theorem5.5.2: Let be an orthonormal
basis for an inner product space .
If , then .
pf:
1 nu u
V
1
n
i ii
v c u
,i ic u v
1 1
1
, , ,
n n
i i j j j i jj j
n
j ij ij
u v u c u c u u
c c
Cor: Let be an orthonormal basis for an inner product space .
If and , then .
pf:
V
1 nu u
1
n
i ii
u a u
1
n
i ii
v b u
1
,n
i ii
u v a b
1 1
5.5.2
1
, , ,
n n
i i i ii i
nTheorem
i ii
u v a u v a u v
a b
Cor: (Parseval’s Formula)
If is an orthonormal basis for an
inner product space and , then
pf: By Corollary 5.5.3,
1 nu u
V1
n
i ii
v c u
2 2
1
n
ii
v c
2 2
1
,n
ii
v v v c
Example 4:
and form
an orthonormal basis for .
If , then
and
1
1
21
2
u
2
1
21
2
u
21 2
2
xx
x
1 2 1 21 2
5.5.21 2 1 2
1 2
, , ,2 2
2 2
Theorem
x x x xx u x u
x x x xx u u
2 22 2 21 2 1 2
1 22 2
x x x xx x x
Example 5: Determine without computing
antiderivatives .
sol:
xdx
4sin
24 2 2 2
2
0
24 2
2 2
sin sin ,sin sin
1 cos 2 1 1 1sin cos 2
2 22 2
1and ,cos 2 is an orthonormal set of ,
2
sin sin
1 1 3
2 42
xdx x x x
xx x
x C
xdx x
Def: is said to be an orthogonal matrix if the column vectors of form an
orthonormal set in .
Example 6:
The rotational matrix
and the elementary reflection matrix
are orthogonal matrix .
n nQ
nQ
cossin
sincos
cossin
sincos
Properties of orthogonal matrices:
If is orthogonal, thenn nQ
1
2 2
( ) The column vectors of form an orthonormal
basis for .
( )
( )
( ) , , preserve inner product
( ) preserve norm
( ) preserve angle
n
i Q
ii Q Q I QQ
iii Q Q
iv Qx Qy x y
v Qx x
vi
Theorem 5.5.6: If the columns of form an orthonormal set in , then and the least squares solution to is
This avoid computing matrix inverse .
m
m nA IAA
Ax b
1x A A A b A b
Theorem 5.5.7 & 5.5.8:
Let be a subspace of an inner product
space and let . Let be
an orthonormal basis for .
If , where ,
then
1 2, , , nx x x
S
x V
(i)
(ii) - - in .
p x S
y x p x y p S
VS
1
n
i ii
p c x
, for each i ic x x i
The vector is said to be t projectionhe of onto .p x S
Cor5.5.9:
Let be a subspace of and
If be an orthonormal basis for
and then
the projection of onto is .
pf:
1 ku u
S
S
1 ,kU u u
m
Sb
p
p UU b
1 1 2 2
1 1
2 2
From Thm.5.5.8, ... ,
where
Therefore, .
k k
T
TT
Tk k
p c u c u c u Uc
c u b
c u bc U b
c u b
p UU b
.mb
Note: Let columns of be an
orthonormal set
1 kU u u
1
1
1
k
k
k
i ii
u
p UU b u u b
u
u b u
(i) The projection of onto ( ) is the
sum of the projection of onto each .
(ii) The matrix is called the projection
matrix onto .
i
T
b R U
b u
UU
S
Example 7: Let
Find the vector in that is closet to
Sol: 5,3,4 .w
, ,0 , .S x y x y p
S
1 2
1 0
Clearly, , is a basis for . Let 0 1 ,
0 0
Thus,
1 0 0 5 5
0 1 0 3 3
0 0 0 4 0
e e S U
p UU w
1 1
2 21 1
: , what is ?2 2
0 0
THW U UU
Approximation of functions
Example 8: Find the best least squares approximation to
on by a linear function .
Sol:
xe 1,0
2
2( )
12 2
0
( . ., Find g( ) 0,1 ( ) min ( ) ,
where , .)
x x
p x Pi e x P e g x e p x
f f f f dx
2
1
0
1 2
(i) Clearly, 1, 0,1 ,but 1, is not orthonormal.
(ii) seek a function of the form , ( ) 1
1 1 1, ( ) 0
2 21 1
2 12
1 1, 12( ) for
2
span x P x
x a x a
x a x a dx a a
x
u u x
2m an orthonormal set of 0,1 .P
Sol:
1
1 1 0
1
2 2 20
1 1 2 2
(iii) c , 1
c , 3 3
Thus, the projection
( )
1 ( 1) 1 3(3 )( 12( ))
2
x x
x x
u e e e
u e u e dx e
p x c u c u
e e x
(4 10) 6(3 )
is the best linear least square approximation to on 0,1 .x
e e x
e
Approximation of trigonometric polynomialsFACT: forms an orthonormal set in with respect to the inner product
Problem: Given a continuous 2π-periodic function , find a trigonometric polynomial of degree n
which is a best least squares approximation to .
1,cos ,sin
2nx nx n N
,0 C
dxxgxfgf )()(
1,
)(xf
0
1
( ) cos sin2
n
n k kk
at x a kx b kx
)(xf
Sol: It suffices to find the projection of onto
the subspace
The best approximation of
has coefficients
)(xf
1,cos ,sin 1, ,
2span kx kx k n
( )nt x
0
1 1, ( )
2 2
1,cos ( )cos
1,sin ( )sin
k
k
a f f x dx
a f kx f x kxdx
b f kx f x kxdx
Example: Consider with inner product of
(i) Check that is orthonormal
(ii) Let
,0 C
dxxgxfgf )()(
2
1,
0, 1, ,ikxe k n
1( )
21
( )2
nikx
n kk n
ikxk
k k
t c e
c f x e dx
a ib
Similarly, k kc c
(iii)
(iv)
cos sin
ikx ikxk k
k k
c e c e
a kx b kx
0
1
cos sin2
nikx
n kk n
n
k kk
t c e
aa kx b kx
5-6 Gram-Schmidt Orthogonalization Process
Cram-Schmidt Orthogonalization ProcessQuestion: Given an ordinary basis ,
how to transform them into an orthonormal
basis ?
1 2, ,..., nx x x
1 2, ,..., nu u u
Given ,Clearly
Clearly,
Similarly,
Clearly, We have the next result
1 nx x
1 11
1u x
x 1 1{ } { }span u span x
1 2 1 1 2 2 12 1
1, , ( )p x u u u x p
x p
1 2 1 2 1 2& { , } { , }u u span x x span u u
2 3 1 1 3 2 2
3 3 23 2
, ,
1( )
p x u u x u u
and u x px p
3 1 3 2 1 2 3 1 2 3, & { , , } { , , }u u u u span x x x span u u u
1u
1p
2x
Theorem5.6.1: (The Gram-Schmidt process)
H. (i) Let be a basis for an inner
product space .
(ii)
C. is an orthonormal basis. 1 nu u
1 nx x
V
1 11
1 11
11
1,
1, 1, , 1
,
K K KK K
K
K K j jj
u xx
u x p K nx p
where p x u u
Example: Find an orthonormal basis for with
inner product given by
, where
Sol: Starting with a basis
3P
),()(,3
1i
ii xgxPgP
.1&0,1 321 xxx
2,,1 xx
1 2 1
1 2
11 1 1 11 1
Let , ,..., be the projection vectors defines in Thm. 5.6.1, and
let , ,..., be the orthonormal basis of ( ) derived from the
Gram-Schmidt process.
Define
n
n
kk
p p p
q q q R A
r a a r q
r
1 for 2,...,
and for 1,..., 1 by the Gram-Schmidt process.
k k
Tik i k
a p k n
r q a i k
Theorem5.6.2: (QR Factorization)
If A is an m×n matrix of rank n, then A
can be factored into a product QR, where Q
is an m×n matrix with orthonormal columns
and R is an n×n matrix that is upper triangular
and invertible.
Proof. of QR-Factorization
1 2 1
1 2 1
11 1
Let , ,..., be the projection vectors defined in Thm.5.6.1,
and let , ,..., be the orthonormal basis of ( ) derived from
the Gram-Schmidt process.
Define
n
n
kk k k
p p p
q q q R A
r a
r a p
1
1 11 1
2 12 1 22 2
1 1
and for 1,... -1 for 2,...,
By the Gram-Schmidt process,
Tik i k
n n
r q a i kk n
a r q
a r q r q
a r q
... nn nr q
Proof. of QR-Factorization (cont.)
1 2
11 12 1
22 2
If we set ( , ,..., ) and define to be the upper triangular matrix
0 ,
0 0
then the th column of the product wi
n
n
n
nn
Q q q q R
r r r
r rR
r
j QR
1 1 2 2
1 2
ll be
... for 1,... .
Therefore,
( , ,..., )
j j j jj j j
n
Qr r q r q r q a j n
QR a a a A
Theorem5.6.3:
If A is an m×n matrix of rank n, then the
solution to the least squares problem
is given by , where Q and R are the
matrices obtained from Thm.5.6.2. The solution
may be obtained by using back substitution to solve .
Ax b
1x̂ R Q b
x̂
ˆRx Q b
Proof. of Thm.5.6.3
ˆLet be the solution to the leaset squares problem
ˆ
ˆ
ˆ ( ) ( ) ( )
ˆ ( )
TAT T
T T
T T
I
x
Ax b
A Ax A b
QR QRx QR b QR Factorization
R Q Q R
1
ˆ ( is invertible)
ˆ ˆ or
T T
T T T
T
x R Q b
R Rx R Q b R
Rx Q b x R Q b
Example 3: Solve
By direct calculation,
1
2
3
1 2 1 1
2 0 1 1
2 4 2 1
4 0 0 2bA
x
x
x
R
Q
QRA
200
140
125
1
2
2
4
2
4
1
2
4
2
2
1
5
1
1
1
2
Q b
The solution can be obtained from
5 2 1 1
0 4 1 1
0 0 2 2