e 600 chapter 1: introduction to vector spaces · previewintroduction the algebraic structure of...
TRANSCRIPT
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
E 600
Chapter 1: Introduction to Vector Spaces
Simona Helmsmueller
August 12, 2018
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Goals of this lecture:
• Understand formal mathematical thinking and notation,
including the difference between properties and definitions
• Know the concepts of span, linear independence, basis and
dimension
• Gain an intuition for open and closed sets, continuity and
convergence
• Be able to graphically illustrate convex sets, a convex hull and
the separating hyperplane theorem
Following lectures (both in this class and other
courses) will assume these goals have been reached!
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Contents
Introduction
The Algebraic Structure of V.S.
Definition
Subspaces and Linear Dependence
Normed V.S. and Continuity
Norms
Open sets, Closed sets, Compact sets
Continuity
Convex sets and the separating hyperplane theorem
Convex sets
Planes, halfspaces and the Separating Hyperplane Theorem
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Introduction
Main objective of the theory of vector spaces:
Geometrical insights at hand with 2-or 3-dimensional real vectors
are really helpful. Can we, in some way, generalize these insights to
other mathematical objects, for which a geometric picture is not
available?
IN THIS CHAPTER, BY VECTOR WE NEED NOT MEAN A
N-TUPLE OF REAL NUMBERS, BUT MAY REFER TO MANY
MORE OBJECTS (FUNCTIONS, SEQUENCES, MATRICES,...)!
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Introduction
Main objective of the theory of vector spaces:
Geometrical insights at hand with 2-or 3-dimensional real vectors
are really helpful. Can we, in some way, generalize these insights to
other mathematical objects, for which a geometric picture is not
available?
IN THIS CHAPTER, BY VECTOR WE NEED NOT MEAN A
N-TUPLE OF REAL NUMBERS, BUT MAY REFER TO MANY
MORE OBJECTS (FUNCTIONS, SEQUENCES, MATRICES,...)!
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Contents
Introduction
The Algebraic Structure of V.S.
Definition
Subspaces and Linear Dependence
Normed V.S. and Continuity
Norms
Open sets, Closed sets, Compact sets
Continuity
Convex sets and the separating hyperplane theorem
Convex sets
Planes, halfspaces and the Separating Hyperplane Theorem
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Real Vector Space)
Let X := (X,+, ·) be a set of vectors with two operations: the
vector addition + : X× X 7→ X and the scalar multiplication
· : X× R 7→ X. X is called a vector space if
(i) Vector addition and scalar multiplication are closed operations:
∀x, y ∈ X, λ ∈ R x + y ∈ X and λ · x ∈ X
(ii) Vector addition is commutative: ∀x, y ∈ X x + y = y + x
(iii) Vector addition is associative: ∀x, y, z ∈ X
x + (y + z) = (x + y) + z
(iv) There exists a null element 0 in X such that: ∀x ∈ X
x + 0 = x
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition
(v) Scalar multiplication is associative: ∀λ, µ ∈ R ∀x ∈ X
λ(µx) = (λµ)x
(vi) Scalar multiplication is distributive over vector and scalar
additions:
∀λ ∈ R ∀x, y ∈ X λ(x + y) = λx + λy
∀ λ, µ ∈ R ∀ x ∈ X (λ+ µ)x = λx + µx
(vii) If 1 denotes the scalar multiplicative identity and 0 the scalar
zero, then:
∀x ∈ X 1x = x and 0x = 0n
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Exercise:
Does the following define a vector space?
for [a, b] ⊂ R, define V := {f : [a, b]→ [a, b]},
∀f ∈ V , a ∈ R : af := f : [a, b]→ [a, b] with f (x) := af (x)
and
∀f , g ∈ V : f + g := h : [a, b]→ [a, b] with h(x) := f (g(x))
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Cartesian product)
Let X := (X,+, ·) and Y := (Y,+, ·) be two real vector spaces.
We define the cartesian product of X and Y, denoted X×Y as the
collection of ordered pairs (x , y) with x element of X and y
element of Y together with two operations: addition and scalar
multiplication, defined respectively as
(x1, y1) + (x2, y2) = (x1 + x2, y1 + y2) and λ(x , y) = (λx , λy).
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Dot product)
Let x = (x1, ..., xn), y = (y1, ..., yn) ∈ Rn. Then the dot product of
these two n-dimensional vectors is a real number:
x • y = x1 · y1 + ...+ xn · yn.
Example:
If x = (1, 2, 3) and y = (5, 6, 7) then
x • y = 1 · 5 + 2 · 6 + 3 · 7 = 38.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Exercise:
Pick three vectors u, v ,w ∈ R4 and a scalar λ. Verify that the
following properties hold and discuss with your neighbor why this is
so.
Theorem(Properties of the dot product)
(a) u • v = v • u(b) u • (v + w) = u • v + u • w(c) u • (λv) = λ(u • v) = (λu) • v(d) u • u ≥ 0
(e) u • u = 0→ u = 0
(f) (u + v) • (u + v)) = u • u + 2(u • v) + v • v
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Closure Under an Operation)
Let X := (X,+, ·) be a real vector space. We say that Y ⊆ X is
closed under the addition if and only if, for any two elements y1and y2 in Y, we have that y1 + y2 belongs to Y. Similarly, we can
define closure under scalar multiplication.
Definition(Vector Subspace)
Let X := (X,+, ·) be a real vector space and Y a non empty subset
of X. We say that Y is a subspace of X if and only if Y is closed
under vector addition and scalar multiplication.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Closure Under an Operation)
Let X := (X,+, ·) be a real vector space. We say that Y ⊆ X is
closed under the addition if and only if, for any two elements y1and y2 in Y, we have that y1 + y2 belongs to Y. Similarly, we can
define closure under scalar multiplication.
Definition(Vector Subspace)
Let X := (X,+, ·) be a real vector space and Y a non empty subset
of X. We say that Y is a subspace of X if and only if Y is closed
under vector addition and scalar multiplication.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem( Intersection and Addition of Subspaces)
Let M and N be subspaces of a real vector space X. Then their
intersection, M ∩ N, is a subspace of X.
What about the union?
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem( Intersection and Addition of Subspaces)
Let M and N be subspaces of a real vector space X. Then their
intersection, M ∩ N, is a subspace of X.
What about the union?
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem(Generated Subspace (a.k.a Span))
Let Y be a subset of a real vector space X. Then, the set
Span(Y), which consists of all vectors in X that can be expressed
as linear combinations of vectors in Y, is a subspace of X. It is
called the subspace generated by Y or span of Y and it is the
smallest subspace which contains Y.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Examples:
Let Y1 = {(1, 0), (0, 1)}. What is Span(Y1)?
Let Y2 = {(1, 0), (0, 2), (0, 0.5)}. What is Span(Y2)?
Let Y3 = {f (x) = x + 1, g(x) = x2 + 2}. What is Span(Y3)?
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Example:
Let u, v ∈ Rn. Then, Span(u, v) = {λu + µv : λ, µ ∈ R}. If u is a
multiple of v , then Span(u, v) is simply the line spanned by u, and
Span(u, v) = Span(u) = Span(v). However, if u is not a multiple
of v , then Span(u, v) is a two-dimensional plane, which contain
the lines Span(u) and Span(v).
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Linear Dependence, Linear Independence)
Let x be an element of a real vector space X. x is said to be
linearly dependent upon a set S of vectors of X if it can be
expressed as a linear combination of vectors from S. Equivalently,
x is linearly dependent upon S if and only if x ∈ Span(S). If that
is not the case, the vector x is said to be linearly independent of
the set S. Finally, a set of vectors is said to be a linearly
independent set if each vector of the set is linearly independent of
the remainder of the set.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem(Testing Linear Independence)
A necessary and sufficient condition for the set of vectors
x1, x2, ..., xn to be linearly independent is that:
Ifn∑
k=1
λkxk = 0, then ∀k = 1, 2, ..., n λk = 0.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Example (from Simon & Blume (1994)):
The vectors
e1 =
1
0...
0
, ..., en =
0...
0
1
∈ Rn
are linearly independent, because if c1, ..., cn ∈ R such that
c1e1 + ...+ cnen = 0,
c1
1
0...
0
+ c2
0
1...
0
+ ...+ cn
0
0...
1
=
c1
c2...
cn
=
0
0...
0
.The last vector equation implies that c1 = c2 = ... = cn = 0.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Basis and Space Dimension)
A finite set S of linearly independent vectors is said to be a basis
for the space X if S generates X. The number of elements in the
basis of vector space X is called its dimension.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem(Uniqueness of the Dimension)
Any two bases for a finite dimensional vector space contain the
same number of elements.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Exercise:
What is the dimension of a plane in Rn?
What is the dimension of the set of all real-valued functions
defined on [a, b]?
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Contents
Introduction
The Algebraic Structure of V.S.
Definition
Subspaces and Linear Dependence
Normed V.S. and Continuity
Norms
Open sets, Closed sets, Compact sets
Continuity
Convex sets and the separating hyperplane theorem
Convex sets
Planes, halfspaces and the Separating Hyperplane Theorem
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Normed Space)
Let X be a vector space. If we can define a real-valued function ‖.‖which maps each element x in X into a real number ‖x‖, and if
that function is such that:
(i) ∀x ∈ X, ‖x‖≥ 0, ‖x‖= 0 if and only if x = 0,
(non-negativity)
(ii) ∀x , y ∈ X, ‖x + y‖≤ ‖x‖+‖y‖, (triangle
inequality)
(iii) ∀x ∈ X ∀λ ∈ R, ‖λx‖= |λ|‖x‖. (absolute
homogeneity1)
Then ‖.‖ is called a norm for X and (X, ‖.‖) a normed space.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
An important example for us is the Euclidean norm in Rn:
∀x = (x1, ..., xn) ∈ Rn ‖x‖:=
(n∑
i=1
x2i
)1/2
= (x ′x)1/2
The Euclidean norm is important (e.g. OLS) and intuitive
(geometric interpretation). All following results and definitions
make use of the Euclidean norm. However, you should bear in
mind that there are more general concepts of norms and the
following definitions can be generalized to suit these. Read the
lecture notes for the general version.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
An important example for us is the Euclidean norm in Rn:
∀x = (x1, ..., xn) ∈ Rn ‖x‖:=
(n∑
i=1
x2i
)1/2
= (x ′x)1/2
The Euclidean norm is important (e.g. OLS) and intuitive
(geometric interpretation). All following results and definitions
make use of the Euclidean norm. However, you should bear in
mind that there are more general concepts of norms and the
following definitions can be generalized to suit these. Read the
lecture notes for the general version.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(ε-Open Ball)
Let (X, ||·||) be a Euclidean space, x0 be an element of X, and ε be
a strictly positive real number. The ε-open ball Bε(x0) centered at
x0 is the set of points whose distance from x0 is strictly smaller
than ε, that is:
Bε(x0) = {x |x ∈ X, ||x − x0||) < ε}.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(ε-Closed Ball)
Let (X, ||·||) be a Euclidean space, x0 be an element of X, and ε be
a strictly positive real number. The ε-closed ball Bε[x0] centered
on x0 is the set of points whose distance from x0 is smaller than or
equal to ε, that is:
Bε[x0] = {x |x ∈ X, ||x − x0||≤ ε}.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Interior Point, Interior)
Let A be a subset of a metric space X. The point a in A is said to
be an interior point of A if and only if there exists ε > 0 such that
the ε-open ball centered at a lies entirely inside A. The collection
of all interior points of A is called the interior of A, denoted Int(A)
or A.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Open Set)
Let A be a subset of a metric space X. A is said to be an open set
if and only if A =Int(A).
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Closure Point, Closure)
Let A be a subset of a metric space X. The point x in X is said to
be a closure point of A if and only if, for every ε > 0, the ε-open
ball centered at x contains at least one point a that belongs to A.
The collection of all closure points of A is called the closure of A,
denoted A.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Closed Set)
Let A be a subset of a metric space X. A is said to be a closed set
if an only if A = A.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Boundary Point, Boundary)
Let A be a subset of a metric space X. The point x in X is said to
be a boundary point of A if and only if, for every ε > 0, the ε-open
ball centered on x contains at least one point a that belongs to A
and at least one point ac that belongs to the complement of A,
AC. The collection of all boundary points of A is called the
boundary of A and denoted ∂A.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem(Properties of Open Sets)
Let (X, d(., .)) be a metric space. Then
(i) ∅ and X are open in X.
(ii) A set A is open if and only if its complement is closed.
(ii) The union of an arbitrary (possibly infinite) collection of open
sets is open.
(iii) The intersection of a finite collection of open sets is open.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem(Properties of Closed Sets)
Let (X, d(., .)) be a metric space. Then
(i) ∅ and X are closed in X.
(ii) A set A is closed if and only if its complement is open.
(iii) The union of a finite collection of closed sets is closed.
(iv) The intersection of an arbitrary (possibly infinite) collection of
closed sets is closed.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Continuous Function)
A function mapping from space X to Y is continuous at x0 ∈ X if
and only if, for every ε > 0, there is a δ > 0 such that if
||x − x0||< δ, then ||f (x)− f (x0)||< ε. A function that is
continuous at every point of its domain is said to be continuous.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Convergence)
An infinite sequence of vectors {xn}n∈N in X is said to converge to
a vector x ∈ X iff the sequence {||xn − x ||}n∈N of real numbers
converges to 0. That is,
∀ε > 0 ∃N ∈ N ∀n > N ||xn − x ||< ε.
In this case, we write xn →n→∞ x .
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Exercise: Show that the limit of a converging sequence is unique
in a metric space.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem(Characterization of Continuity)
A function mapping from X to Y is continuous at x0 ∈ X if and
only if xn → x0 implies f (xn)→ f (x0).
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Contents
Introduction
The Algebraic Structure of V.S.
Definition
Subspaces and Linear Dependence
Normed V.S. and Continuity
Norms
Open sets, Closed sets, Compact sets
Continuity
Convex sets and the separating hyperplane theorem
Convex sets
Planes, halfspaces and the Separating Hyperplane Theorem
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Convex Combination)
A convex combination of the vectors x1, x2,...,xn is a linear
combination of the vectors, i.e., a sumn∑
i=1
λixi , λi ∈ R, such that
the following additional requirements hold:
n∑i=1
λi = 1 and ∀i λi ∈ [0, 1]
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Exercise:
Let n = 2 and x , y ∈ R with x < y . Graphically display the convex
combination of these x and y .
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Characterization of Convex Sets)
Let Y be a subset of a vector space X. Y is convex if and only if
the convex combination between any two of its elements is
contained in Y.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Convex Hull)
Let Y be the subset of a vector space X. The convex hull, denoted
Co(Y) is the smallest convex set containing Y.
The convex hull of Y may also be expressed as the set of all
possible convex combinations of the elements of Y:
Co(Y) =
{x ∈ X : ∃y1, y2, · · · , yn ∈ Y and λ ∈ [0, 1]n
s.t.n∑
i=1
λi = 1 and x =n∑
i=1
λiyi
}
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
TheoremLet u, v be vectors in Rn. In the plane spanned by the two vectors,
let θ be the angle between them (see picture below). Then
u • v = ||u||·||v ||cosθ.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Hyperplane)
Let X be a subspace of Rn. Then, a hyperplane of X is a set of the
form:
Hba := {x ∈ X | a • x = b}
where a is an element of Rn that is different from 0 and b is an
element of R.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Example: Lines
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Example: Planes
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Definition(Halfspace)
Let X be a subspace of Rn. Then, a halfspace of X is a set of the
form:
Hb−a := {x ∈ X | a′x ≤ b}
or
Hb+a := {x ∈ X | a′x ≥ b}
where a is an element of Rn that is different from 0 and b is an
element of R.
Preview Introduction The Algebraic Structure of V.S. Normed V.S. and Continuity Convex sets and the separating hyperplane theorem
Theorem(Separating Hyperplane Theorem)
Let C and D be two convex sets in a metric space X. Further,
assume C∩D = ∅. Then, there exists a 6= 0 in Rn and b in R such
that for all x in C a′x ≤ b and for all x in D a′x ≥ b. The
hyperplane {x ∈ X | a′x = b} is called a separating hyperplane for
the sets C and D.