math 504 fall 2016 notes week 3, lecture 1home.ku.edu.tr/.../notes_files/slides_week3_1.pdf · math...

Math 504 Fall 2016 NotesWeek 3, Lecture 1

Emre Mengi

Department of MathematicsKoç University

Istanbul, Turkey

Emre Mengi Week 3, Lecture 1

Outline

Orthogonal Rank 1 Representation

Column Space, Rank, Null Space in terms of SVD

Matrix Norms

It follows from the SVD

A = UΣV ∗

u(1) . . . u(m)],

v (1) . . . v (n)],

Σ = diag(σ1, σ2, . . . , σp)m×n

A = σ1u(1)[v (1)

]∗+ σ2u(2)

[v (2)

]∗+ · · ·+ σpu(p)

[v (p)

]∗where p := min{m,n}.

A = σ1u(1)[v (1)

]∗+ σ2u(2)

[v (2)

]∗+ · · ·+ σpu(p)

[v (p)

]∗Rank

(u(j) [v (j)]∗) = 1 for j = 1, . . . ,p.(

u(j) [v (j)]∗)∗ u(k) [v (k)]∗ = 0, whenever j 6= k .

Example

[1 −33 −1

[1/√

2 −1/√

21/√

2 1/√

] [4 00 2

] [1/√

2 −1/√

21/√

2 1/√

[1/√

21/√

] [1/√

2 −1/√

+ 2[−1/√

21/√

] [1/√

2 1/√

[2 −22 −2

[−1 −1

]Emre Mengi Week 3, Lecture 1

A = σ1u(1)[v (1)

]∗+ σ2u(2)

[v (2)

]∗+ · · ·+ σpu(p)

[v (p)

]∗Rank

(u(j) [v (j)]∗) = 1 for j = 1, . . . ,p.(

Example

[1 −33 −1

[1/√

2 −1/√

21/√

2 1/√

] [4 00 2

] [1/√

2 −1/√

21/√

2 1/√

[1/√

21/√

] [1/√

2 −1/√

+ 2[−1/√

21/√

] [1/√

2 1/√

[2 −22 −2

[−1 −1

A = σ1u(1)[v (1)

]∗+ σ2u(2)

[v (2)

]∗+ · · ·+ σpu(p)

[v (p)

]∗Rank

(u(j) [v (j)]∗) = 1 for j = 1, . . . ,p.(

Example

[1 −33 −1

[1/√

2 −1/√

21/√

2 1/√

] [4 00 2

] [1/√

2 −1/√

21/√

2 1/√

[1/√

21/√

] [1/√

2 −1/√

+ 2[−1/√

21/√

] [1/√

2 1/√

[2 −22 −2

[−1 −1

A = σ1u(1)[v (1)

]∗+ σ2u(2)

[v (2)

]∗+ · · ·+ σpu(p)

[v (p)

]∗Rank

(u(j) [v (j)]∗) = 1 for j = 1, . . . ,p.(

Example

[1 −33 −1

[1/√

2 −1/√

21/√

2 1/√

] [4 00 2

] [1/√

2 −1/√

21/√

2 1/√

[1/√

21/√

] [1/√

2 −1/√

+ 2[−1/√

21/√

] [1/√

2 1/√

[2 −22 −2

[−1 −1

Column and Null Spaces, Rank in terms of SVD

Column Space in terms of SVD

Sort the singular values of A ∈ Cm×n from the largest to thesmallest. (Below p := min{m,n}.)

σ1 ≥ σ2 ≥ · · · ≥ σr > σr+1 = · · · = σp = 0

If all singular values are nonzero, then r = p and

σ1 ≥ σ2 ≥ · · · ≥ σp > 0.

Col(A) = {Ax | x ∈ Cn}= span{u(1),u(2), . . . ,u(r)}

In other words, {u(1),u(2), . . . ,u(r)} is an orthonormalbasis for Col(A).

σ1 ≥ σ2 ≥ · · · ≥ σr > σr+1 = · · · = σp = 0

σ1 ≥ σ2 ≥ · · · ≥ σp > 0.

σ1 ≥ σ2 ≥ · · · ≥ σr > σr+1 = · · · = σp = 0

σ1 ≥ σ2 ≥ · · · ≥ σp > 0.

Rank in terms of SVD

Rank in terms of SVDRank(A) = # of nonzero singular values.

Example

10 54 −32 14

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 5

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 5√

3 > 0, so

{u(1), u(2)

01/√

3−1/√

an orthonormal basis for Col(A). Furthermore, Rank(A) = 2.

Example

10 54 −32 14

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 5

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 5√

3 > 0, so

{u(1), u(2)

01/√

3−1/√

Example

10 54 −32 14

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 5

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 5√

3 > 0, so

{u(1), u(2)

01/√

3−1/√

Example

10 54 −32 14

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 5

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 5√

3 > 0, so

{u(1), u(2)

01/√

3−1/√

Example

10 54 −32 14

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 5

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 5√

3 > 0, so

{u(1), u(2)

01/√

3−1/√

Null Space in terms of SVD

Null space of A ∈ Cm×n is defined by

Null(A) := {x ∈ Cn | Ax = 0}.

Example

[1 −11 −1

], Null(A) = span

Proposition

Null(A) is a subspace of Cn.

Null(A) := {x ∈ Cn | Ax = 0}.

Example

[1 −11 −1

], Null(A) = span

Proposition

Null(A) := {x ∈ Cn | Ax = 0}.

Example

[1 −11 −1

], Null(A) = span

Proposition

σ1 ≥ σ2 ≥ · · · ≥ σr > σr+1 = · · · = σp = 0

σ1 ≥ σ2 ≥ · · · ≥ σp > 0.

Null(A) = {x ∈ Cn | Ax = 0}= span{v (r+1), v (2), . . . , v (n)}

In other words, {v (r+1), v (2), . . . , v (n)} is anorthonormal basis for Null(A).

σ1 ≥ σ2 ≥ · · · ≥ σr > σr+1 = · · · = σp = 0

σ1 ≥ σ2 ≥ · · · ≥ σp > 0.

Null(A) = {x ∈ Cn | Ax = 0}= span{v (r+1), v (2), . . . , v (n)}

In other words, {v (r+1), v (2), . . . , v (n)} is anorthonormal basis for Null(A).

Null Space in terms of SVDExample

6 80 06 8

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 00 0

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 0, so{v (2)}=

{[4/5−3/5

]}an orthonormal basis for Null(A).

Corollary (Rank-Nullity)

For a matrix A ∈ Cm×n, we have

Rank(A) + dim Null(A) = n.

6 80 06 8

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 00 0

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 0, so{v (2)}=

{[4/5−3/5

6 80 06 8

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 00 0

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 0, so{v (2)}=

{[4/5−3/5

6 80 06 8

2 1/√

3 1/√

√3 −2/

2 −1/√

3 −1/√

2 00 00 0

[ 3/5 4/54/5 −3/5

We have σ1 = 10√

2, σ2 = 0, so{v (2)}=

{[4/5−3/5

Matrix Norms

Common Matrix Norms

Let A ∈ Cm×n.

Frobenius norm

‖A‖F :=

√√√√ m∑j=1

n∑k=1

|ajk |2

2-norm‖A‖2 := max

x∈Cn,‖x‖2=1‖Ax‖2

p-norm (p ∈ R, p ≥ 1)

‖A‖p := maxx∈Cn,‖x‖p=1

‖Ax‖p

Common Matrix Norms

Let A ∈ Cm×n.

Frobenius norm

‖A‖F :=

√√√√ m∑j=1

n∑k=1

|ajk |2

x∈Cn,‖x‖2=1‖Ax‖2

p-norm (p ∈ R, p ≥ 1)

‖Ax‖p

Common Matrix Norms

Let A ∈ Cm×n.

Frobenius norm

‖A‖F :=

√√√√ m∑j=1

n∑k=1

|ajk |2

x∈Cn,‖x‖2=1‖Ax‖2

p-norm (p ∈ R, p ≥ 1)

‖Ax‖p

Characterizations

For a matrix A ∈ Cm×n of the form

a(1) a(2) . . . a(n)]

[a(1)]T[a(2)]T

...[a(m)

we have

‖A‖1 = max{‖a(1)‖1, ‖a(2)‖1, . . . , ‖a(n)‖1},‖A‖2 = σ1 (largest singular value of A),

‖A‖∞ = max{‖a(1)‖1, ‖a(2)‖1, . . . , ‖a(m)‖1}

Definition and Properties

A matrix norm ‖ · ‖ : Cm×n → R just like a vector norm satisfiesthe following properties.

(i) ‖A‖ > 0 for all nonzero A ∈ Cm×n

(ii) ‖αA‖ = |α|‖A‖ for all α ∈ C and all A ∈ Cm×n

(iii) ‖A + B‖ ≤ ‖A‖+ ‖B‖ for all A,B ∈ Cm×n

Additionally, most commonly used matrix norms on Cn×n alsosatisfy the following submultiplicative property.

‖AB‖ ≤ ‖A‖‖B‖ for all A,B ∈ Cn×n

Definition and Properties

A matrix norm ‖ · ‖ : Cm×n → R just like a vector norm satisfiesthe following properties.

(i) ‖A‖ > 0 for all nonzero A ∈ Cm×n

(ii) ‖αA‖ = |α|‖A‖ for all α ∈ C and all A ∈ Cm×n

(iii) ‖A + B‖ ≤ ‖A‖+ ‖B‖ for all A,B ∈ Cm×n

Additionally, most commonly used matrix norms on Cn×n alsosatisfy the following submultiplicative property.

‖AB‖ ≤ ‖A‖‖B‖ for all A,B ∈ Cn×n

Invariance of the 2-norms

Let U ∈ Cm×m and V ∈ Cn×n be unitary matrices.(i) For every v ∈ Cm, we have ‖Uv‖2 = ‖v‖2.

(ii) For every A ∈ Cm×n, we have ‖UAV‖2 = ‖A‖2.

math 504 fall 2016 notes week 3, lecture 1home.ku.edu.tr/.../notes_files/slides_week3_1.pdf · math...

Documents