1 d. r. wilton ece dept. ece 6382 introduction to linear vector spaces reference: d.g. dudley,...

1

D. R. Wilton ECE Dept.

ECE 6382 ECE 6382

Introduction to Linear Vector Spaces

Reference: D.G. Dudley, “Mathematical Foundations for Electromagnetic Theory,” IEEE Press, 1994.

Fields Fields

F A is a set of mathematical objects with rules for adding,

subtracting, multiplying, and dividing and with the usual associativity,

commutivity,and distributivity properties.

field

{ 0 ,1,2, }

{ , 2, 1,0,1,2

(

,

)

}

Fields :

- rationals (ratios of integers)

- real numbers

- complex numbers

a field :

- natural numbers,

- integers,

Q

R

N

Z

C

Not

{ 0 ,1,2, }

{ , 2, 1,0,1,2

(

,

)

}

Fields :

- rationals (ratios of integers)

- real numbers

- complex numbers

a field :

- natural numbers,

- integers,

Q

R

N

Z

C

Not

Note it is implied that applying the operation rules of addition or multiplication

to elements of the set results in elements of the set. (I.e., the operations

map elements of the set onto itse .) lf

1

1

0

( ) 0

( ) 1

Field properties :

associativity,

commutativity

existence of an

identity element

existence of inverse

distributivity

1

1

0

( ) 0

( ) 1

Field properties :

associativity,

commutativity

existence of an

identity element

existence of inverse

distributivity

Linear Vector SpacesLinear Vector Spaces

a b c

Given a collection of , , , , in a set and a set of

scalars, , , , defined on a of real numbers or complex

numbers , we say is a linear vector space if the operations

RF

vector objects

field

S

S

)

) .

,

)

i

ii

iii

a b c a b c

0 a 0 0 a a

0 a 0 0 a a a

a

of

addition and multiplication are defined with the following rules :

There exists a vector in such that

( : )

For each in ,there exists a ve

S

S

S

Addition:

,

)iv

a

a a a a 0

a a a a a 0 a

a b b a

ctor in such that

( )

S

S : S

- "there exists"

- "such that"

- "in" or "a member of"

- "for all"

- "therefore"

- "because" or "s

:

ince"

Linear Vector Spaces, cont’dLinear Vector Spaces, cont’d

)

)

)

1)

v

vi

vii

viii

a a

a a

a b a b

a a a

Multiplication by a scalar :

a b c

, , ,

S

( ) ( ),

( )

Operators :

F

, , ,

,

a b a b

Valid on a vector or pair of vectors always yields a vector

operations

in the space, i.e., S S

Linear Vector Spaces, cont’dLinear Vector Spaces, cont’d

F

0

a

b

a

a b

Field

Linear vector space

b

A linear vector space enables us to form linear combinations of vector objects.

S

Linear Vector Space ExamplesLinear Vector Space Examples

3

1 2 3

1 2 3

1 1 2 2 3 3

ˆ ˆ ˆ ( , , )

( , , , , ) ,

( , , , , ), ,

( , , , , ),

(

x y z x y z

N N

N

N k k

N N

a x y z a

a

b

a b

a

Ordinary 3 -D vectors

or

N- dimensional Euclidean (unitary) space, ( )

( )F R

R

R C

=

Example :

Example :

1 2 3, , , , )

C( , )

( ), ( ) ( ), ( ) ( , ),

( ) ( ),

( )

N

f g f g

f g

f

a b

a b

a

where are functions oncontinuous

Example :

Linear Vector Space Examples , cont’dLinear Vector Space Examples , cont’d

11 12 1 11 12 1

21 22 2 21 22 2

1 2 1 2

11 11 12 12 1 1

21 21 22 22 2 2

1 1 2 2

, ,

MN

N N

N N

M M MN M M MN

N N

N N

M M M M

M N

a b

a b

, matrices of dimension MExample :

11 12 1

21 22 2

1 2

,

N

N

MN MN M M MN

a

Linear IndependenceLinear Independence

1 2 3

1

1

21

, , , , ,

0 1,2, , .

0

.

(1,2), (

N

N

k k kk

k

N

k kk

N

k N

x x x x

x 0

x 0

a b

vectors, are if the only

solution of is for all

The vectors are if some can be found

such that

In ,

independent

dependent

R

Example :

1 2

3,6)

3 . (1,2) (3,7) a b 0 a b

are dependent since

However, and

are independent.

DimensionalityDimensionality

3

dim

1

ˆ ˆ ˆ, ,

N N

N

x y z

A linear space has if it posesses a set of independent

vectors, whereas every set of vectors is dependent.

In , the vectors are easily shown to be indep

Example :

S S

R

dimension

dim 3

ˆ ˆ ˆ ˆ ˆ ˆx y z x y za a a a a a

k

a

a x y z a x y z 0

endent. The dimension must be

three ( ) since any other vector can be written in terms of these, i.e.,

( )

If we find a set of independent vectors for

S

eve 1,2,

dim

( ) 2 sin , 1,2, , , (0,1)k k

k k

f k k N

f

( ), then has

( )

The set is of infinite dimension, as shown

on the following slide.

Example :

S

S

ry infinite

dimension .

Linear Independence and DimensionalityLinear Independence and Dimensionality

0 0

1

0

0,1 ( ) 2 sin , 1,2, , , (0,1)

( ) 0 0

0,( ) ( )

1,

k k

N N

k k k kk k

k k

f k k N

f

kf f d

k

f

f

In , the functions are

independent. To show this, examine the expression

(or ).

Since , multipl

Example :

C( )

( )

(0,1)

0, 1,2, , ( )

( ) 1,2, ,

( )

k

k

k

f

N f

f k

f

ying both sides by

and integrating over the interval yields

are independent.

In this case the set for is , i.e. each

can be put into

countably infinite

dim

a one - to - one correspondence with the set of

integers; the associated vector space is thus of infinite dimension

( ). The set is then said to be independent if finite subset

is inde

S every

pendent. The analysis above shows that is the case here.

BasesBases

1 2

1

, ,

.

N

k

N

k kk

x S

x x x

x

x x

A set is a for if the vectors in the set are independent, and if every

vector can be written as a linear combination of the ; namely we can write

This representation

basis S

1

k k

N

k kk

k

x x

is unique, as we can show by contradiction. I.e., we assume

non-uniqueness so that there must exist non- vanishing coefficients (for

at least one ) such that

.

Subtracting these las

1

0

0

,

N

k k kk

k

k k

k k

k

k

x

x

t two equations yields

,

which, in view of the independence of , implies

.

Thus for all contradicting our hypothesis above.

Note: If N is finite and dim S = N,then “and if” in the first line above may be replaced by “then”. I.e., any N independent vectors form a basis. Unfortunately, it is not the case that any infinite set of independent vectors forms a basis when dim S = ∞!

Bases, cont’dBases, cont’d

1 2

1 2

1

, ,

. 1 , , ,

0 0.

N

N

N

k kk

N

N

x

x x x

x x x x x

x x

If is - dimensional, any linearly independent set is a

Indeed, let Then by definition, the vectors must

be linearly dependent,

,

Solving for

basis.S

S

1

1 2, , ,

Nk

kk

N

x x

x x x

, we have

,

and hence the set is a basis.


1 2

1

2

1

, ,

(1,0,0, ,0)

(0,1,0, ,0)

(0,0,0, ,1)

(

N

N

Nk

N

e e e

e

e

e

e

a

a

The vectors where

form a basis in . First, the can be easily be shown to be

linearly independent. Secondly, for any ,

Example :

R

R

2

1 2

1 1 2 2

31 2

1

2

3

, , , )

(1,0,0, ,0) (0,1,0, ,0) (0,0,0, ,1)

, ,

ˆ(1,0,0)

ˆ(0,1,0)

ˆ(0,0

.

,1)

N

N

N N

N

e e e

e e e

e x

e y

e z

I.e., any vector in the space

can be written as a linear combination of In

R


1 2

1 2 1

, ,

1

, ,

N

N

NN N

N

N

N

e e e

a a a a

is - dimensional.

- First, there exists independent vectors, .

- Second, we show that any set of vectors is dependent.

Let be a set of vectors , i, n . We

R

R

Example :

1

1

( )

1

, 1, 2, , 1

, 1,2, , 1

m

N

m mm

m k

Nm

m k kk

m N N

m N N

a 0

a e

a e

must show

there exists , not all zero, such that

.

We express the 's as a linear combination of the bases :

Substituting into th

,

,

e ab1

( )

1 1

1( )

1

,

0, 1,2, ,

N Nm

m k kk m

Nm

m k km

k N

e 0

e

ove and reversing the summation order yields

since are independent


(1) ( 1)1 1 11

( )

1 (1) ( 1)1

0

0, 1,2, ,

0

1

N

Nm

m km N

N N N

k N

N N

The above is an underdetermined system of linear

equations, and hence has at least one non- trivia

1 2 1

1

, ,m N N

N N

a a a a

l solution. Therefore

at least one coefficient is non- zero. Hence every set

is linearly dependent and is - dimensional. (Note tha

,

t )

,

= .R R

Inner Product SpacesInner Product Spaces

( , ) ,

, , *

, , ,

x y x y

x y y x

x y z x z y z

A linear space is a if for every ordered pair

of vectors in there exists a unique scalar in , denoted ,

such that

a.

b.

c.

S

complex inner product space

, , ,

, 0,

*

x y x y

x x x 0

d. with equality if and only if ("iff")

For a we drop the conjugate ( ) in (a) and

require in (c). We usually assume a complex inner product

real inner product space,

space.

F

0

x

y

S

, x y ,

,

if is a complex inner product space

if is a real inner product spaceF

S

S

Field

Inner product space

The inner product is a generalizationof the dot product of vectors in R3

Inner Product Spaces, cont’dInner Product Spaces, cont’d

, 0 0

, * ,

, , *

, *

* , *

* ,

0 y

x y x y

x y y x

y x

y x

x y

, which follows from choosing in (c).

Proof :

Example :

Example :


1 2 3 1 2 3

1

1 1

,

( , , , , ) , ( , , , , ), ,

, *

, * *

,

*

N

N N k k

NN

k kk

N N

k k k kk k

a b

a b

a b

a b

N- dimensional (unitary) space :

is an inner product space if we define

since

a.

Example :

C

C

1

1 2 3

1

1 1

1 1

* * , *

( , , , , )

, *

* * , ,

, * * , ,

N

k kk

N

N

k k kk

N N

k k k kk k

N N

k k k kk k

b a

c

a b c

a c b c

a b a b

.

b. If ,

c. .


2

1 1

2

1 1

,

, * 0

, * 0 0,

(0,0, ,0)

(0,0, ,0) , 0

N N

k k kk k

N N

k k k kk k

k

a a

a a

a a

a 0

a 0 a a

d. First, note that is non- negative since

.

Second, note that if

,

Finally, if

Example (cont'd) :


C( , )

( ) , ( ) ( ), ( ) ( , ),

, , ( ) * ) .(

f g f g

f g f g d

a b

a b

where are functions on

Define That this forms a complex inner

product space is proved similarly to the previous case,

Example :

continuous

with integration

replacing summation.

Since the inner product generalizes the notion of a dot product ofvectors in R3, we often read <a,b> as “a dot b” and say that <a,b> is a “projection of a along b” or vice versa.

The Cauchy-Schwarz-Bunjakowsky (CSB) The Cauchy-Schwarz-Bunjakowsky (CSB) InequalityInequality

2

2

, , ,

, , ,

, 0,

, 0

,

,

, , ,, * ,

, ,

x y x x y y

x y x x y y

x y

y 0 y y

x y

y y

x y x y y xy x x y

y y y y

or

To prove, note first that if the result is automatically true,

so assume and define

Hence

2, y y .

The Cauchy-Schwarz-Bunjakowsky (CSB) The Cauchy-Schwarz-Bunjakowsky (CSB) Inequality, cont’dInequality, cont’d

22

2

22

,, * , ,

,

0 , , , , * ,

,, , , ,

,

x yy x x y y y

y y

x y x y x x y x y y x y

x yx x x y x x y y

y y

Next note that by property (d) and the above result,

Orthogonality and OrthonormalityOrthogonality and Orthonormality, 0

, 1,2,

, 0, , 0,

, ,

k

i j i i i i

i j ij i j i ij

k

i j C

C

x y x y

z

z z z z z 0

z z z z

A pair of vectors and are if .

A set of vectors is an orthogonal set if

. ( if )

It is an ortho set if

(

orthogonal

normal

1

,

1,

0,

,

ij

N

k kk

i j

i j

0

z 0

if only orthogonal)

where is the .

An orthogonal set that does not contain the vector is ,

and can be shown to be independent. Indeed, forming

and

Kronecker delta

proper

1

, , , 0

0, 1,2, ,

, 1,2,

i

N

k k i i i i ik

i

k

i N N

k

z

z z 0 z z z

z

taking the inner product with on both sides,

if is finite. Otherwise, the

is linearly independent. countably infinite set

0

Note that vector is orthogonal to the vector!

every

y

x

Normed Linear SpaceNormed Linear Space

1 2 1 2

,

0

x

x

x x 0

x x

x x x x

is a if, for every vector

there is a unique number such that

a. , with equality if and only if

b. ,

c. (triangle inequalit

normed linear space

.

F

S S

,

,

x x x

x y x y

y)

Though many possible norms exist, we focus on the

,

.

The Cauchy - Schwarz -Bunjakowsky (CSB) inequality can thus be written as

.

Note that if one wishe

norm induced by the

inner product

,

,cos 1 .

x y

x y

x y

s, one can now define the between two vectors as

Thus, the and allow us to generalize the notions of

and or

angle

norm inner product magnitude

angle between 3dinary vectors in . R

0

x

yS

xy

x y

x y

Normed Linear Space, cont’dNormed Linear Space, cont’d

1 2 1 2

0

x x 0

x x

x x x x

To check that the norm induced by the scalar product satisfies the norm

conditions, namely,

a. , with equality if and only if ,

b. , ,

c. (triangle ineq

F

,

, * ,

x x x

x x x x x x

uality) ,

we first note that since , property (a) is true by inner product property (d) .

Property (b) above is true since .

Finally, for property (c) above, take two ve2

2 2

2 2

2 2 2

,

, , , , ,

, , , * ,

2Re ,

2 , Re ), ,

2 ( )

x y

x y x y x y x x x y y x y y

x x x y x y y y

x x y y

x x y y x y x y

x x y y x y

x y x y

CSB

ctors and note that

(since )

S

0

x

yS

x y

Normed Linear Space, cont’dNormed Linear Space, cont’d

2 2

2 2

1 2

2

1

, ,

, , , ,

, , , ,

2 2

( , , , )

,

NN

N

kk

x y x y x y x y x y x y

x x y x x y y y

x x y x x y y y

x y

a

a a a

C

A useful identity is

(parallelogram law)

For the norm is

(Pytha

Example :

Example :

2

( ) ( , )

, ( )

f

f f f f d

C

gorean theorem)

For the norm is

Example :

y

x

x y

x y

Convergence of a SequenceConvergence of a Sequence

0

x y x y

x y

x y

Normed spaces allow us to measure the "closeness" between two vectors. Since

,

it is reasonable to say and are "close" if

.

We need this notion of closeness, for example, to discuss th

1

0

. lim

k kk

k

k kk

N

k N

x x

x x x

x x x x

S

e convergence

of a sequence. We say a sequence of vectors , to a

vector if , given an , there exists a number such that

whenever We write or , and

converges

0

0, :

k

k kN k N

x x

x x x x

note that .

( if , given an )

Continuity of the Inner ProductContinuity of the Inner Product

1k kk

x x x

In approximating vectors, we require the notion of of an inner product,

which is the idea that if a sequence of vectors converges to ,the

continuity

S

Example : Continuity of the Inner Product

2 2 2

, , lim , ,

, ,

.

0

.

., 0

,

k kk

k k

k k

k k

x h x h x h x h

h

x h x h x x h

x x h x x h

x x x x

n

( )

where is any vector in

Note it is sufficient to show that

We have, by the CSB inequality, that

But since a

S

, 0

lim

.

, lim , ,

k

k kk k

k

x x h

x h x h x h

x

nd hence

Note the result is equivalent to

(I.e., "The limit of projections of is the projection of the limit.")

Convergence in the Cauchy SenseConvergence in the Cauchy Sense

1

,

0

min ( , )

lim 0

k kk

m n

m nm n

N

m n N

x x

x x

x x

A sequence of vectors if,

given an there exists a number such that whenever

, that is,

.

If the sequence converges t

converges in the Cauchy senseS

2 2m n m n m n

x

x x x x x x x x x x

x

o , then it also converges in the Cauchy

sense because

.

Unfortunately, the reverse may not be true since there is no guarantee

the limit is in . What must bS e done is to enlarge the definition of the set

to include its limit points to "complete" the set. A normed linear space

is thus if every Cauchy convergence converges to a

member of .

complete

S

S S

S

Convergence in the Cauchy Sense, cont’dConvergence in the Cauchy Sense, cont’d

1

p q p q

The real number system may be constructed by first including

all the rationals, i.e. numbers of the form where and are integers.

As we know, however, there are many irra

=

tional n

umb

Example :

R

2 ,eers (e.g., etc.).

Indeed, any neighborhood of an irrational number contains an infinite number

of rationals that are arbitrarily close to it. A Cauchy sequence of these can be

constructed

, ,

that converges to the irrational number. Unfortunately, however,

the limit is not in the set of rationals making up the elements of the sequence!

The solution is to append the irrationals to the set of rationals so that every

Cauchy sequence of elements from the new set converges to an element

of the set. This is the sense in which the real numbers are "complete."

0 1 2 3 4 512

11

7x 2

36

7x

1122

7x

x

kx


,

1,1

0, 1 0

( ) , 0 1 , 1,2,

1, 1 1

1 1

,

,

kf k k k

k

The normed space is incomplete. The following well known

sequence in

converges in the Cauchy sense, but not to an element of . To

s

C

C

C

Example :

0, 1 0

( ) , 0 1, 1,2,

1 , 1 1

0, 1 1

m k

m k

m k mf f m k

k m k

k

ee this, choose two members of the sequence and form the difference;

for this is

Convergence in the Cauchy Sense, cont’dConvergence in the Cauchy Sense, cont’d1.0

kfmf

1

k

1

m

11

2 2 2

1 0

2

1

1max

k

m k m k m k

m k

f f f f d f f dk

k m

f f

Thus we have the (admittedly crude, but sufficient) estimate (refer to the figure)

(= area of )

For we reverse the argument and finally obtain

1,

0, 1 0

1, 0 1

0

m

k m

f

so that , the unit step function. But the step function

is not continuous at , so the limit function is not in the original space of

continuous functions. Clearly, the space of continuous functions is too

restrictive to be complete in the norm induced by the inner product.


0

Note, as seen above, several versions of the step function with different values

at can result from different limiting processes!

Later we will find that these different step functions, which diff

0 er in value

only at the point , should, in a certain sense, be regarded as the same

function!

1.0

kf

mf

1

k1

m

0, 1 0

1, 0 1mf

,

1.0

kfmf

1

k

1

m

0.5

0, 1 0

1 2, 0

1, 0 1mf

0 An alternative sequence with a different limit at ... 0

... and still another sequence with a different limit at .

Hilbert Spaces Hilbert Spaces

1 2

1 2

( , , , )

( , , , )

N k k kk N

N

a

a

A linear vector space is a if it is complete in the norm

induced by the inner product.

is a Hilbert spacesince a sequence of vectors

converges to if

H

R

Hilbert space

.lim

.

ki i

kk Ni i

N

a

and only if Since both

and must be real by the completeness of the reals, then must bein

is a Hilbert spacesince the above argument may be applied to the

real

d

an

R

C

22

1

, *

k

k k kk k k i i i

i i

a

a a a

imaginary parts separately.

and are Hilbert spaces, however, only if, for all the elements of

the sequence, the norm induced by the inner product is finite,

R C

1

.

k

aWe usually say that the vectors in or must have components

that are .

R C

square summable

2 The set of square - summable sequences in is called . C

Hilbert Spaces, cont’d Hilbert Spaces, cont’d

, The space of continuous functions is incomplete as we have seen from

our example of a sequence of continuous functions leading to the step function.

Completion of this space leads to the s

C

2

2 2

,

, ( )

, ( ) *( ) ( ) .

f

f f f f f d f d

pace of functions that are

on the interval , namely functions such that

It should be noted, that in order to complete

this

L square

integrable

space, first the definition of

integration must be extended. In particular, we do not simply interpret the integral

in the traditional Riemann sum sense, but rather as a so - called Lebesgu

. Secondly, even the notion of equality of two functions must be extended.

We won't discuss these technicalities, which arise only rarely in applications,

except to mention that

e

integral

if two functions differ at only a number or

number of points, they are treated as equivalent, and their Lebesgue

integrals are equal. Thus, for example, all unit st

finite countably

infinite

( )

0.

u

ep functions are treated as

equivalent no matter what (finite) value is assigned to function at the discontinuity,

2, , LThe set of square - integrable functions on is called .

Linear SubspacesLinear Subspaces

, ,

, 1,2, ,k k N

x y

x y

x

F

M S

M M

M

, a subset of , is a or , provided that ,

is in whenever and are in .

is itself a linear space.

Let be linearly independ ent

Example

linear subspace linear manifold

1

2 , ,

N

k kk

N

x

x y

M

R

vectors in the Hilbert space . Define the set

of all linear combinations of the vectors,

.

Then is a linear subspace.

In let be vectors in the first quadrant. This is

Example

n

x y

a linear subspace since it is easy

to find and such that is in the first quadrant.

ot

not

Linear Subspaces, cont’dLinear Subspaces, cont’d

1 ˆ ˆ ˆ3 2 x x y z

2 ˆ ˆ ˆ0.1 4 2 x x y z

1 2 x x x

31 2

3

31

, dim 2

.

x x

x

O

S =R M M =

R

S R

.

=

• In the example shown, the linear combinations

of in form a linear subspace of

The linear subspace is a (2 -D) passing through in

• Linear combinations of in

form

plane

3

dim 1

dim dim

O

M =

R .

M S

a linear subspace of that is a (1-D)

passing through in

• In general, linear combinations of a set of independent vectors form a linear subspace

of that is a thro

line

hyperplane O Sugh in .

• The set of vectors that generates a given linear subspace is not unique!

1x

2x

x

2x

1x

x

z

y

Example

O

Gram-Schmidt OrthogonalizationGram-Schmidt Orthogonalization

1

1

N

k k

N

k k

N

x

e

M H

Let be a sequence of independent vectors that generate

a linear subspace . The process is a

constructive procedure for generating an orthonormal sequence

from the indepe

Gram - Schmidt

11 1 1

1

22 2 2 1 1 2

2

33 3 3 1 1 3 2 2 3

3

1 1 2 2 1 1

,

, ,

, , ,

, 1,2, ,n n n n n n n

nn

n

n N

zz x e

zz

z x x e e ez

zz x x e e x e e e

z

z x x e e x e e x e ez

ez

ndent

.

vectors :

Let and

Let and

Let and

I

.

n general,

a

.

.nd

Note

1

1

N

k kN

k k

ez

that while the sequence is orthonormal, the sequenceis orthogonal.

Gram-Schmidt Orthogonalization, cont’dGram-Schmidt Orthogonalization, cont’d

1

1 11

N

k k kkN

N N

k k k kk kk

k

e x

x x e

z 0

Any linear combination is also a linear combination of ,

; i.e., and generate the same linear subspace.

The Gram - Schmidt process may terminate early on

ly if for

1

N

k kk N

xsome ; this can happen if and only if the sequence is linearly independent.

The above property can thus be used as a test to determine if a sequence of vectors contains dependencies or n

not

1k k kk

z z z

In a numerical Gram - Schmidt process, once should become suspicious of dependencies if or, say, if for some . Determining

if there are dependencies or near - dependen

ear - dependencies.

cies in data is a common and important data processing task. Hence it should not be surprising that variants

of the Gram - Schmidt procedure are ubiquitous in numerical processing, especially in linear analysis!


12

2 321

1

1 1 1

1

1, , , , 1,1

, ( ) ( )

1 11 2

2(1)

x x x

f g f x g x dx

z edx

Given the sequence , we use the Gram - Schmidt

process, with inner product , to produce an

orthogonal sequence

.

Let .

Example :

L

1 12 2

1 12 2

2 2 1 2

12 2 2 2

32

23 1 22

1

,2 2 3 2

,2 2 , 3 2 1 31 3

45 8 1 31 3

xz x x x e x

x dx

z x x x x xx

e xx dx

Let

Let

Et

.

.

c. ...


1 0

2 1

2 23 2

34 3

4 25 4

56 5

1( ) 1

23 2 ( )

145 8 1 3 ( ) 3 1

21

( ) 5 321

( ) 35 30 381

( ) 638

e P x

e x P x x

e x P x x

e P x x x

e P x x x

e P x x

The sequence so generated is proportional to the orthogonal Legendre polynomials,

370 15

(1) 1k

x x

P

etc. ...

The Legendre polynomials are orthogonal but not ortho ; they

are normalized such that .

normal

Closed SetsClosed Sets

,

M

M

C( )

A linear subspace is if it contains all limits of sequences

constructed from members of .

The space is a linear subspace since a sum of two continuous

functions is conti

Example

closed

, C( )

nuous. However, we showed we can construct a sequence

of continuous functions that converges to a discontinuous function. Hence

is not closed.

If a closed linear subspace is conta

Example

1k k

k

x

x x x

M H

M M

M H H M M

M

ined in a Hilbert space, , then

is itself a Hilbert space. Indeed, let be a Cauchy sequence in .

Then since , . But since isclosed, and therefore

is a Hilbert space.

Best Approximation in a Hilbert SpaceBest Approximation in a Hilbert Space

1

1

M

k k

M

M k k Mk

k

N M N

M

x z

x

x x z x

Let be a Hilbert space of dimension , and , be

an orthonormal set in . Consider approximating by the finite, term sum

We want to choose the coefficients

o as

,

,

s

H

H

M H

2

2

1 1

,

, , , ,

, ,

M M

M M M

M M M M

M M

k k j jk j

x x x x x

x x x x x x

x x x x x x x x

x x z z

to obtain the "best" approximation

to . Specically, we want to minimize the norm . We expand as

follows :

1 1

1 1 1 1

2

1 1 1

, ,

, * , , * ,

, , * * ,

M M

k k k kk k

M M M M

k j k j k k k kk j k k

M M M

k k k k kk k k

kj

z x x z

x x z z z x x z

x x x z x z

Best Approximation in a Hilbert Space, cont’dBest Approximation in a Hilbert Space, cont’d

2 2

1 1 1

2

1 1

2 2

1 1

, , * * ,

, , , * ,

, , ,

M M M

M k k k k kk k k

M M

k k k k kk k

M M

k k kk k

x x x x x z x z

x x x z x z x z

x x x z x z

,

and since all the terms above are positive, the errror

i

s

min

1

,k k

M

k k

x z

x

z

imized if we choose

,

the associated with the expansion of in the orthonormal

set .

Fourier coefficients


1

,

, , , ,

M M

M

k kk

j

M j j k k jk

M

ε x x

x x z z

z

ε z x z x z z z

The error associated with the - term expansion may be defined as

Note that the projection of the error on every member of the set vanishes

:

1

1

1

, , , , 0

, , 0.

M

M

j k kj j jk

M

M M k M kk

x z x z x z x z

ε x ε z

;

hence we have the remarkable result


,

.

M

M

M M

M M

x x

ε

x x ε

ε x

In summary, the vector has been decomposed into a vector plus

an error vector namely,

where the error is orthogonal to the approximation vector

H M

.k

x

As the figure suggests, minimizing

the error is equivalent to making it

orthogonal to the linear subspace

generated by the orthogonal

vectors Mε

1x

2x

x

z

y

x

Mx

1 2,x x

M linear subspace

generated by


1

1

1

1

2 2 2( ) ( )

2 1 2 1 2 1

2 1( ), ,

2

, ( )

k k k

k k k k

P x P x dxk k

kP x

f g f x

z z z

Example

The Legendre polynomials satisfy

( ) .

Though not orthonormal, renormalization yields the ortho functions

where

normal

1

1

1

( ) .

( 1,1)

( )

2 1 2 1, , ( ) ( ) ( )

2 2

M

k k Mk

k k k k

g x dx

f x

f x

k kf P x f x P x dx

x

x z x

x z

Hence the "best" approximation to , is

where we choose

,

and where "best" implies minimizing the error in the norm indu

21

1 11

( ) ( ) ( )M M

M k k k kk k

f x f x f x dx

x z z

ced by the inner product,


1

0

0

sin (0, )

sin sin2

2sin , ,

, ( ) ( ) .

Nn

mn

n m n mn

nx x

mx nx dx

nx

f g f x g x dx

z z z

Example

The trigonometric functions are orthogonal on :

,

and may be renormalized to yield orthonormal functions,

,

where

1 1

0

(0, )2

( ) sin

2 2, , sin ( )sin

N N

n n n Nn n

n n

f x

f x nx

f nx f x nx dx

x

x z x

x z

Hence the "best" approximation to , is

where we choose

,

where again "best" is in terms of minimizing the error in the n

;

.N

n

x x

z

orm induced by the inner product.

Generally, we do not yet know if to answer that question requires the additional notion

of "completeness" of the set But in this case, we recognize this as a

2

0

( ) .N f x x

x x

Fourier sine series, for

which we know that if do


2

1

( ) (0,1)

1,( ) , 1, ,

0, otherwise

,

( )

ˆ ( ) ( )

n nN N

n

m n mn

n nn

f x

xx n N

N

f x

f x x

y

Consider the piecewise constant approximation of . The sequence

is orthogonal, .

We want to approximate as

Example

L

1

1

1

1

1

0

1

( ), , ( ) ( ) ,

( ),

( ),

nNn nN N

nnNN

nN

N

n n

nn N

n n

x f g f x g x dx

f x dxf

f x dxdx

yWith and we thus have

1.0

1.0

x

( )f x

n

1nN n

N

( )n n x

Orthogonal Complement to a Linear SubspaceOrthogonal Complement to a Linear Subspace

e

x

A vector is a member of the set , the

to , if it is orthogonal to every vector in .

: Let be any vector in the Hilbert space , and

let

H M

M M

H

M

orthogonal complement

The projection theorem

0 0

0

y x x y x y y

y

be a closed linear subspace. Then there exists a unique vector

closest to in the sense that for all in . The

necessary and sufficient condition that is the unique minim zin

i

H

M

0 0

0

x y y x

e x y x

g vector

is that is in .The vector is called the of onto ;

is the of onto .

M M

M

projection

projection

The Projection TheoremThe Projection Theorem

x

z

y

x

0y

M linear subspace

0 e x y

M

M

linear subspace

(orthogonal to )

The Projection Theorem and Best The Projection Theorem and Best Approximation Approximation

1

1

ˆ

ˆ

Mj j

M

j jj

j

y y

y y

y y

H H H Let , and be a linearly independent set of vectors in . We

form the sum

and seek to approximate by with a suitably chosen set of coefficients .

Since lin

M

M H M M

M

ear combinations like the above form a linear subspace , and the subspace

is closed by virtue of the fact that ,the limit of sequences in must be in ,

and hence is closed. It therefore

1,2, ,

ˆ( ), 0, 1,2, ,

ˆ

,

j

k

j j k

j M

k M

y

y y y

y

y y

M,

meets the requirements of the projection

theorem. Since , the projection theorem yields

Substituting the series for an

d rearranging yields

1

, , 1,2, ,M

kj

k M

y y

The Projection Theorem and Best The Projection Theorem and Best Approximation, cont’d Approximation, cont’d

1

1 1 2 1 1 1 1

1 2 2 2 2 2 2

1 2

, , , 1,2, ,

, , , ,

, , , ,

, , , ,

M

j j k kj

M

M

M M M M M M

k M

y y y y

y y y y y y y y

y y y y y y y y

y y y y y y y y

The system of equations,

may be written in matrix for

m as

ˆj

y where the are solutions of the matrix equation and determine above.

The matrix is the transpose of the so - called , which also

appears in proofs on the independence of

Gram matrix

j

j

y

y

the set . The result generalizes

the Fourier coefficient result and reduces to it if the are orthogonal. I.e.,

the matrix reduces to a diagonal matrix in this case, and the solution is trivi

2

, ,

,j j

jj j j

y y y y

y y y

al :


2

1

1

( ) (0,1)

1

Nn

n

f x

x

N

yConsider the polynomial approximation of . The sequence

is linearly independent since, by the fundamental theorem of

algebra, the only solution of the degree polynomial

Example

L

1

1

0, (0,1)

1 0

0

Nn

nn

n

a x x

N x a

x

equation

are its roots; hence the only solution that holds for is .

(Alternatively, set in the equation and all its non- vanishing derivatives.)

We want to approx

all

1

1

( )

ˆ ( )N

nn

n

f x

f x x

imate by


1

0

1

0

1 1

0

( )

( )

( )

1

2

11

0

1 12 1

0 0

1 12

1 1 12 3 1

1 1 11 2 1

, , ( ) ( ) ,

1, , , ( )

1

1

N

f x dx

f x xdx

f x x dxN

nn

m n mm n m

N

N

N N N

x f g f x g x dx

x dx f x x dxm n

y

y y y y

With and we thu s have

Solution of the matrix equation yields the coefficients of the best approximation.

But notice that, for example, the next - to - last and last columns (rows) approach

one another for la 1 1 1 1 1 11 2 2 1 2 1

.

N N N N N NN

N

rge , the resulting

near - dependencies in row / column entries make it more difficult to obtain accurate

coefficients for large

;


1

1

1

1

2

1

1

11

( ) (0,1)

, ( , ), 2, ,

( ) , ( , ), 1, , 1

0, otherwise

1,2, ,

n

n n

n

n n

x xn nx x

x xn n nx x

nn N

f x

x x x n N

x x x x n N

x n N

y

Example

Consider the piecewise - linear approximation of . The sequence

where ,

L

1

1

( ) 0

0

( )

ˆ ( ) ( )

N

n nn

n n

N

n nn

a x

a x x

f x

f x x

is linearly independent

since the only solution of

is , as is easily seen by evaluating it at .

To approximate by

1.0

1.0

x

( )f x

nx

( )n n x

n

1nx 1nx


1

0

1 1

0 0

1 13 6

1 2 16 3 6

1 26 3

1

( ), , ( ) ( ) ,

, ( ) ( ) , , ( ) ( )

0 0

01

0 01

0 0 0

n n

m n m n m m

x f g f x g x dx

x x dx f x x dx

N

y

y y y y

With and we thus have

Evaluation of the basis interaction matrix yields the matrix equation

1

10

1

20

1

30

1

0

1

2

3

( ) ( )

( ) ( )

( ) ( )

( ) ( )3N

N

f x x dx

f x x dx

f x x dx

f x x dx

n

,

the solution of which yields the coefficients of the best approximation.

Operators in Hilbert SpaceOperators in Hilbert Space

x

y x

y x

x

A is a mapping that assigns to a vector another vector

. We usually write

.

The of the operator is the set of vectors for which the mapping is

defined. Th

S

S

linear operator

domain

L

L

L

1 1 2 2 1 1 2 2( )

y

x

x x x x

e of the operator is the set of vectors resulting from the

mapping.

The operator is if it operates on vectors in a linear vector space

and

A linear oper

S

range

linear

L L L

x x

x

ator with domain is if there exists a real number

such that

for all .

D H

D

boundedL

L

L

Operators in Hilbert Space, cont’dOperators in Hilbert Space, cont’d

11 12 1

21 22 2

1 2

1 2

1 2

1 1 2 2 1 1 2 2 1( )

N

N

N N NN

TN

TN

a a a

a a aA

a a a

A

A

A A A

x

y x

y

x x x x y y

Example

The matrix

is a linear operator thatassigns to the vector

where . is a linear operator since

2

1Ax y

.

The concept of inversion can be generalized to linear operators

and is formally written as

exactly as for matrix equations.


1 2 3

1 2

( , , , )

(0, , , )

k

RA

a

a

Example

Let be the space of all vectors of finite norm and consisting of a

countably infinite set of real numbers,

where .

The is defined as

.

R. >

. >

R

right - shift operator

2

1

,

1

R R

R kk

A A

A

a a

It is easy to show that is linear. In addition is bounded since

and the least upper bound of

..


2

1

0

1

0

(0,1)

( ) ( , ) ( ) ,

( ) ( , )

u x k x x dx f x

u f

k x x d

Example

On the complex Hilbert space consider the

which we can write in operator form as follows :

where is the linear operator

L

L

L=

L integral equation

1 1 2

0 0( , )

x

k x x dxdx

.

We will show the operator is bounded if

.

This property - - - that the kernel of the integral equation is square -

integrable - - - is called the , and the

.

.

Hilbert - Schmidt property operator

it generates is called a . Hilbert - Schmidt operator


1 1 2

0 0

12 2

0

( , )

( )

k x x dxdx

u f x dx

x x

To show that a Hilbert - Schmidt operator is bounded if

,

we note that

where, by the CSB inequality (regard as merely a fixed parameter and the

integration as an inne

L

L

21 1 1 12 2 2 2 2

0 0 0 0

1 12 2 2

0 0

1 1 2 20 0

( ) ( ) ( , ) ( ) ( , ) ( , )

( , )

( , )

f x u x k x x dx u x dx k x x dx u k x x dx

u u k x x dx dx

k x x dx dx M

u M u

r product),

.

Hence .

Finally, if , then we have

L

L

Continuity of Hilbert OperatorsContinuity of Hilbert Operators

0 0 0

0 0

, .u u u u u u

A linear operator on a domain is if,

given an , there exists a such that, for every

, whenever

L

L

D

D L L

H continuous

u

LD0u

uL0uL

Continuity of Hilbert Operators, cont’dContinuity of Hilbert Operators, cont’d

1

0

0 ( lim ) lim .

n n

n nn n

u

u

u u u

A linear operator with domain is continuous if

and only if for every sequence converging to

,

PROOF :

1.

L

L

L

L D

D

D

L L L

L

H

If is continuous, the operator and limit may be int

0

0 0 ,

n n

n n

u u u

u u n N u u

By continuity, given an we may select a such that

, and since is a member of a convergent sequence,

for all ; hence it is also true that

fo

L L

erchanged.

0 lim .nn

n N

u u

r all or, equivalently,

2.

We do not need and will not prove this part.

L L

LIf the operator and limit may be interchanged, is continuous.

Equivalence of Boundedness and Continuity of Equivalence of Boundedness and Continuity of Hilbert OperatorsHilbert Operators

0

0 0

0

.

.

n n

u

u u u u

u

A linear operator with domain is bounded if

and only if it is continuous

PROOF :

1.

Since is bounded and ,

for all Thus given an

L

L

L

L D

L

L D

L L

D

H

If is bounded, it is also continuous.

0

0

0

n

n

u u

u u

, we may select any

and we will have

when .

2.

We do not need and will not prove this part.

L L

LIf is continuous, it is also bounded.

Unbounded Operator ExampleUnbounded Operator Example

. ,

lim lim

cos, 1,

n

n nn n

n

d dx u u

u u u

n xu n

n

Example

Differential operators are generally unbounded, as in this example.

Let If is bounded, it is also continuous, i.e. for any

We choose

L L

L L L

2,

coslim lim 0 lim 0 0.

lim lim ( si

.

n )

n nn n n

nn n

n xu u

nu n x

so that , and therefore

But does not exist, which contradicts our

assumption that is bounded

L L

L

L

Matrix Representation of Bounded Hilbert Matrix Representation of Bounded Hilbert OperatorsOperators

1

1

{ }

lim

.k k

N

kN k

z

u f

u

u z

Bounded Hilbert operators are uniquely determined by a matrix. Let

be a basis for Let be bounded operator, and let

.

Let us expand in terms of the basis as follows :

L

L

H

1 1 1

lim lim lim

lim

k

N N N

k k k k k kN N Nk k k

kN k

u z z z f

.

Since boundedness implies continuity, and by linearity,

Next, project both sides of the latter equality with the basis set,

yielding

L L L L

<1

, , , 1,2,N

k j jz z f z j

.L

Matrix Representation of Bounded Hilbert Matrix Representation of Bounded Hilbert Operators, cont’dOperators, cont’d

1

1 1 2 1

1 2 2 2

lim , , , 1,2,

, ,

, ,

N

k k j jN k

z z f z j

z z z z

z z z z

By linearity and continuity of the inner product, we have

,

which is essentially an matrix equation, which we write as

L

L L

L L1 1

2 2

1 1

,

,

lim

k

N

k k k kN k k

k

f z

f z

u z z

If this matrix equation can be solved, then the coefficients determine

the complete solution according to

The coefficients

.u

are unique only to the choice of the basis; another

basis choice will yield a different set of coefficients, but the same solution

Non-Negative, Positive, and Positive Definite Non-Negative, Positive, and Positive Definite OperatorsOperators

22

, 0,

, 0, 0

, ,

x x x

x x x

x x c x

Convergence criteria are well - established for , ,

and operators that satisfy

for all (non-negative)

for all in (positive)L

L

L D

L D

L

non - negative positive

positive - definite

,

0,

, ,

c x

x x x x

and (positive - definite)

Non-negative positive, and positive - definite operators are , i.e.,

,

Indeed, if the inner product is real, as are all the operators defined above, th

LD

L L

symmetric

, , * ,

[ , ] ,

x x x x x x

x y x y

e

operator is symmetric since

.

For positive, and positive - definite operators an

with respect to operator can be defined, and it

becomes a Hilb

L L L

L L

energy inner product,

,

,x x x

ert space on . The associated

is .

L LD

L

energy

norm

H

Non-Negative, Positive, and Positive Definite Non-Negative, Positive, and Positive Definite Operators, cont’dOperators, cont’d

1

lim 0

lim

n n

nn

e

n

n

x xc

u u u u

u u

u u u

The following can be established :

For positive - definite operators

.

If a sequence to , , we mean

;

if it to ,we write , and we mean

converges

converges in energy

0

lim , 0

n

w

n n

nn

u u

u u u u

g

u u g

.

If the sequence to , , we mean

that for every

.

converges weakly

H,

Non-Negative, Positive, and Positive Definite Non-Negative, Positive, and Positive Definite Operators, cont’dOperators, cont’d

nu

We note (but do not prove) the following relationships between

different types of convergence can be established :

If is bounded, convergence implies convergence in energy.

Convergence implies weak

L

w

n

w

n n

u f g

u u f

convergence.

Convergence in energy implies where .

If, however, is bounded, then in .

If is positive - definite, convergence in energy impliesconvergence.

LL

L L

H

H

The Moment MethodThe Moment Method

We developed a matrix representation of the operator for a basis defined

on a Hilbert space. The procedure led to an infinite matrix in an infinite

dimensional space. In reality, one often finds an

L

0

u f

u f

u

approximate solution using a

finite number of bases. The method applies to linear operator, but

convergence proofs may not always be available.

To solve consider

.

Approximate using

L f

L

any

1

1

1

1

{ }

{ }

Nk k

N

N k kk

N

Nk k

N

k kk

u

u u u

u u

w

u

a finite set of bases, , in the domain of the operator :

.

Now replace by above and make the residual error orthogonal to a set of

testing functions, :

L , 0, 1, , ,2,mf w m N

The Moment Method, cont’dThe Moment Method, cont’d

1

1 1 2 1 1

1 2 2 2 2

1

, , , 1,2, ,

, , ,

, , ,

,

N

k k m mk

mk k m

N

Nmk

N

u w f w m N

L f

u w u w u w

u w u w u wL

u w

Rearranging and using linearity and inner product properties, we have

.,

or in matrix form,

L

L L L

L L L

L L

1

2

2

,

,,

, , ,

m

N N N N

f w

f wf

u w u w f w

There are no convergence proofs for general operators . For many special

cases and for operators appearing in specific applicatio

L

ns , convergence

proofs may exist in spaces appropriate to the operator; developing such

convergence proofs is an active area of research in the math community.

1 d. r. wilton ece dept. ece 6382 introduction to linear vector spaces reference: d.g. dudley,...

Documents