tensor decompositions, matrix completion and singular...
TRANSCRIPT
![Page 1: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/1.jpg)
Tensor Decompositions, Matrix Completion andSingular Values
Harm Derksen
University of Michigan
ICERM, Computational Nonlinear AlgebraJune 3, 2014
Harm Derksen Tensors, LRMC and Singular Values
![Page 2: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/2.jpg)
Tensor Product Spaces
F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space
Definition
A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)
(v (i) ∈ V (i)).
Problem
Write a given tensor as a sum of the smallest number of puretensors.
(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)
Harm Derksen Tensors, LRMC and Singular Values
![Page 3: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/3.jpg)
Tensor Product Spaces
F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space
Definition
A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)
(v (i) ∈ V (i)).
Problem
Write a given tensor as a sum of the smallest number of puretensors.
(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)
Harm Derksen Tensors, LRMC and Singular Values
![Page 4: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/4.jpg)
Applications
I psychometrics
I chemometrics
I algebraic complexity theory
I signal processing
I numerical linear algebra
I computer vision
I numerical analysis
I data mining
I graph analysis
I neuroscience
I economics/finance
Harm Derksen Tensors, LRMC and Singular Values
![Page 5: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/5.jpg)
Application: Fluorescence Spectroscopy
We have p samples of mixtures of unknown chemical compounds.Every mixture is excited with light of m different wavelengths.Light of n wavelengths is emitted from the mixture . Measuringthe intensities, one obtains an m × n excitation-emisson matrix forevery sample. This yields a p ×m × n matrix, which is a 3-waytensor T .Every chemical compound corresponds to a rank 1 tensor. Bywriting T as the sum of r = rank(T ) pure tensors, we candistinguish r chemical compounds and find the excitation-emissionmatrix for each of them.
Harm Derksen Tensors, LRMC and Singular Values
![Page 6: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/6.jpg)
Application:Fluorescence Spectroscopy(Image: Lei Li, Andrew Barron)
Harm Derksen Tensors, LRMC and Singular Values
![Page 7: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/7.jpg)
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
![Page 8: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/8.jpg)
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
![Page 9: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/9.jpg)
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
![Page 10: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/10.jpg)
Tensor rank
Definition
The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.
For example,
T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1
has rank 2 because
T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1
2(e1−e2)⊗(e1−e2)⊗(e1−e2)
If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.
Harm Derksen Tensors, LRMC and Singular Values
![Page 11: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/11.jpg)
Application: Algebraic Complexity Theory
V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)
Tn =∑n
i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i
rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.
Theorem
If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.
Harm Derksen Tensors, LRMC and Singular Values
![Page 12: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/12.jpg)
Application: Algebraic Complexity Theory
V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)
Tn =∑n
i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i
rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.
Theorem
If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.
Harm Derksen Tensors, LRMC and Singular Values
![Page 13: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/13.jpg)
Application: Algebraic Complexity Theory
Theorem (Strassen 1969)
rank(T2) ≤ 7, so rank(Tn) = O(nlog2(7)) = O(n2.8073...).
Theorem (Williams 2012)
rank(Tn) = O(n2.3727)
Theorem (Masseranti, Raviolo 2013)
rank(Tn) ≥ 3n2 − 2√
2n3/2 − 3n.
Harm Derksen Tensors, LRMC and Singular Values
![Page 14: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/14.jpg)
Low Rank Matrix Completion
Problem
Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.
For example −2 · ·2 3 ·· 6 2
can be completed to a rank 1 matrix−2 −3 −1
2 3 14 6 2
Harm Derksen Tensors, LRMC and Singular Values
![Page 15: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/15.jpg)
Low Rank Matrix Completion
Problem
Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.
For example −2 · ·2 3 ·· 6 2
can be completed to a rank 1 matrix−2 −3 −12 3 14 6 2
Harm Derksen Tensors, LRMC and Singular Values
![Page 16: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/16.jpg)
Low Rank Matrix Completion
Problem
Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.
For example −2 · ·2 3 ·· 6 2
can be completed to a rank 1 matrix−2 −3 −1
2 3 14 6 2
Harm Derksen Tensors, LRMC and Singular Values
![Page 17: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/17.jpg)
Application: The Netflix Problem
The DVD rental company Netflix has 480,189 users, and 17,770movies. The user ratings for every user can be put in a480, 189× 17, 770 matrix A = (ai ,j) for which only few entries areknown (since most users have only seen a fraction of the 17,770movies).
Presumably, the rank of the matrix A is low. Using Low rankmatrix completion, Netflix can predict whether users like a moviethey have seen, and will recommend movies to the users.
Harm Derksen Tensors, LRMC and Singular Values
![Page 18: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/18.jpg)
Reduction LRMC to Tensor Rank
A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank
Define
T =s∑
k=1
eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1
Theorem (D.)
rank(T ) = mrank(A) + s
Harm Derksen Tensors, LRMC and Singular Values
![Page 19: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/19.jpg)
Reduction LRMC to Tensor Rank
A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank
Define
T =s∑
k=1
eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1
Theorem (D.)
rank(T ) = mrank(A) + s
Harm Derksen Tensors, LRMC and Singular Values
![Page 20: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/20.jpg)
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
![Page 21: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/21.jpg)
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
![Page 22: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/22.jpg)
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
![Page 23: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/23.jpg)
Example
A =
(1 t· 1
)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).
mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.
If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.
If t 6= 0 then rank(T ) = 2 and
T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2
Harm Derksen Tensors, LRMC and Singular Values
![Page 24: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/24.jpg)
Rank Minimization Problem (RM)
Problem (Rank Minimization)
Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).
LRMC is a special case of RM where Bk = eik ,jk .
Theorem (D.)
The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.
For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.
Harm Derksen Tensors, LRMC and Singular Values
![Page 25: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/25.jpg)
Rank Minimization Problem (RM)
Problem (Rank Minimization)
Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).
LRMC is a special case of RM where Bk = eik ,jk .
Theorem (D.)
The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.
For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.
Harm Derksen Tensors, LRMC and Singular Values
![Page 26: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/26.jpg)
t1,1,1 t1,2,1 a1,1 a1,2 a1,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0t2,1,1 t2,2,1 a2,1 a2,2 a2,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 1 0 0 λ1,1 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 1 0 0 λ1,2 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 1 0 0 λ1,3 0 0 0 0 0 0 0 0 0 0 0
b1,1 b2,1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0b1,2 b2,2 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0b1,3 b2,3 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 t1,1,2 t1,2,2 a1,1 a1,2 a1,3 0 0 0 0 0 00 0 0 0 0 0 0 0 t2,1,2 t2,2,2 a2,1 a2,2 a2,3 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 1 0 0 λ2,1 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,2 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,3 0 0 00 0 0 0 0 0 0 0 b1,1 b2,1 0 0 0 1 0 0 0 0 00 0 0 0 0 0 0 0 b1,2 b2,2 0 0 0 0 1 0 0 0 00 0 0 0 0 0 0 0 b1,3 b2,3 0 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,3
.
The smallest rank over all ai ,j , bi ,j , λi ,j (1 ≤ i ≤ 2, 1 ≤ j ≤ 3) is12 + rank(T ).
Harm Derksen Tensors, LRMC and Singular Values
![Page 27: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/27.jpg)
Motivation: compressed sensing and convex relaxation
For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.
Problem
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).
But, ‖ · ‖0 is not convex and this optimization problem is difficult,
so instead we consider:
Problem (Basis Pursuit)
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.
Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).
Harm Derksen Tensors, LRMC and Singular Values
![Page 28: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/28.jpg)
Motivation: compressed sensing and convex relaxation
For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.
Problem
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).
But, ‖ · ‖0 is not convex and this optimization problem is difficult,so instead we consider:
Problem (Basis Pursuit)
Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.
Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).
Harm Derksen Tensors, LRMC and Singular Values
![Page 29: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/29.jpg)
Nuclear Norm
V (i) Hilbert space. Instead of the CP model, we consider:
Problem (Convex Decomposition)
Given a tensor T , write T =∑r
i=1 vi for some r and some puretensors v1, . . . , vr such that
∑ri=1 ‖vi‖2 is minimal.
Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.
Definition (Lim-Comon)
The nuclear norm ‖T‖? is the smallest value of∑r
i=1 ‖vi‖2 whereT =
∑ri=1 vi and v1, . . . , vr are pure tensors.
For some tensors we know the nuclear norm but not the rank.
Harm Derksen Tensors, LRMC and Singular Values
![Page 30: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/30.jpg)
Nuclear Norm
V (i) Hilbert space. Instead of the CP model, we consider:
Problem (Convex Decomposition)
Given a tensor T , write T =∑r
i=1 vi for some r and some puretensors v1, . . . , vr such that
∑ri=1 ‖vi‖2 is minimal.
Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.
Definition (Lim-Comon)
The nuclear norm ‖T‖? is the smallest value of∑r
i=1 ‖vi‖2 whereT =
∑ri=1 vi and v1, . . . , vr are pure tensors.
For some tensors we know the nuclear norm but not the rank.
Harm Derksen Tensors, LRMC and Singular Values
![Page 31: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/31.jpg)
Spectral Norm
Definition
The spectral norm is defined by
‖T‖σ = max{|〈T , v〉| | v pure tensor with ‖v‖2 = 1}.
The spectral norm is dual to the nuclear norm, in particular
|〈T ,S〉| ≤ ‖T‖?‖S‖σ
for all tensors S ,T .
Harm Derksen Tensors, LRMC and Singular Values
![Page 32: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/32.jpg)
Example: Determinant Tensor
Consider the tensor
Dn =∑σ∈Sn
sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
Clearly rank(Dn) ≤ n!, actually( nbn/2c
)≤ rank(Dn) ≤ (56)bn/3cn!.
‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1
by Hadamard’s inequality.
‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so
Theorem (D.)
‖Dn‖? = n!
Harm Derksen Tensors, LRMC and Singular Values
![Page 33: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/33.jpg)
Example: Determinant Tensor
Consider the tensor
Dn =∑σ∈Sn
sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
Clearly rank(Dn) ≤ n!, actually( nbn/2c
)≤ rank(Dn) ≤ (56)bn/3cn!.
‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1
by Hadamard’s inequality.
‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so
Theorem (D.)
‖Dn‖? = n!
Harm Derksen Tensors, LRMC and Singular Values
![Page 34: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/34.jpg)
Example: Permanent Tensor
Pn =∑
σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}
Theorem (Carlen, Lieb and Moss, 2006)
max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2
‖Pn‖? =nn/2
n!‖Pn‖?‖Pn‖σ ≥
nn/2
n!〈Pn,Pn〉 = nn/2.
Harm Derksen Tensors, LRMC and Singular Values
![Page 35: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/35.jpg)
Example: Permanent Tensor
Pn =∑
σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.
‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}
Theorem (Carlen, Lieb and Moss, 2006)
max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2
‖Pn‖? =nn/2
n!‖Pn‖?‖Pn‖σ ≥
nn/2
n!〈Pn,Pn〉 = nn/2.
Harm Derksen Tensors, LRMC and Singular Values
![Page 36: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/36.jpg)
Example: Permanent Tensor
Theorem (Glynn 2010)
Pn =1
2n−1
∑δ
(∏ni=1 δi
)(∑n
i=1 δiei )⊗ · · · ⊗ (∑n
i=1 δiei )
where δ runs over {1} × {−1, 1}n−1.
In particular,( nbn/2c
)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so
Theorem (D.)
‖Pn‖? = nn/2
Harm Derksen Tensors, LRMC and Singular Values
![Page 37: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/37.jpg)
Example: Permanent Tensor
Theorem (Glynn 2010)
Pn =1
2n−1
∑δ
(∏ni=1 δi
)(∑n
i=1 δiei )⊗ · · · ⊗ (∑n
i=1 δiei )
where δ runs over {1} × {−1, 1}n−1.
In particular,( nbn/2c
)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so
Theorem (D.)
‖Pn‖? = nn/2
Harm Derksen Tensors, LRMC and Singular Values
![Page 38: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/38.jpg)
t-Orthogonality
Definition
Pure tensors v1, v2, . . . , vr are t-orthogonal if
r∑i=1
|〈vi ,w〉|2/t ≤ 1
for every pure tensor w with ‖w‖2 = 1.
1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.
Theorem (D.)
If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .
Harm Derksen Tensors, LRMC and Singular Values
![Page 39: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/39.jpg)
t-Orthogonality
Definition
Pure tensors v1, v2, . . . , vr are t-orthogonal if
r∑i=1
|〈vi ,w〉|2/t ≤ 1
for every pure tensor w with ‖w‖2 = 1.
1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.
Theorem (D.)
If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .
Harm Derksen Tensors, LRMC and Singular Values
![Page 40: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/40.jpg)
t-Orthogonality
Definition
Pure tensors v1, v2, . . . , vr are t-orthogonal if
r∑i=1
|〈vi ,w〉|2/t ≤ 1
for every pure tensor w with ‖w‖2 = 1.
1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.
Theorem (D.)
If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .
Harm Derksen Tensors, LRMC and Singular Values
![Page 41: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/41.jpg)
Horizontal and Vertical Tensor Product
Theorem (“horizontal tensor product”, D.)
If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.
If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then
V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).
Theorem (“vertical tensor product”, D.)
If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
![Page 42: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/42.jpg)
Horizontal and Vertical Tensor Product
Theorem (“horizontal tensor product”, D.)
If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.
If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then
V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).
Theorem (“vertical tensor product”, D.)
If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
![Page 43: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/43.jpg)
The Diagonal Singular Value Decomposition
Definition
Suppose that (?) : T =∑r
i=1 λivi such that λ1 ≥ · · · ≥ λr > 0and v1, . . . , vr are 2-orthogonal pure tensors of unit length, then(?) is called a diagonal singular value decomposition of T (DSVD).
If d = 2 (tensor product of 2 spaces) then the DSVD is the usualsingular value decomposition. For d > 2, the DSVD is differentfrom the Higher Order Singular Value Decomposition defined byDe Lathauer, De Moor, and Vandewalle. Not every tensor has aDSVD.
Harm Derksen Tensors, LRMC and Singular Values
![Page 44: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/44.jpg)
The Diagonal Singular Value Decomposition
Theorem (D.)
If T has a DSVD then
‖T‖? =∑i
λi , ‖T‖2 =
√∑i
λ2i , ‖T‖σ = λ1
Theorem (D.)
If λ1 > λ2 > · · · > λr then the DSVD is unique.
Theorem (D.)
If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.
Harm Derksen Tensors, LRMC and Singular Values
![Page 45: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/45.jpg)
The Diagonal Singular Value Decomposition
Theorem (D.)
If T has a DSVD then
‖T‖? =∑i
λi , ‖T‖2 =
√∑i
λ2i , ‖T‖σ = λ1
Theorem (D.)
If λ1 > λ2 > · · · > λr then the DSVD is unique.
Theorem (D.)
If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.
Harm Derksen Tensors, LRMC and Singular Values
![Page 46: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/46.jpg)
Example: Matrix Multiplication Tensor
e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal
e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal
Using vertical tensor product, we get
{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}
are 2-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
![Page 47: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/47.jpg)
Example: Matrix Multiplication Tensor
e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal
e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal
Using vertical tensor product, we get
{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}
are 2-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
![Page 48: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/48.jpg)
Example: Matrix Multiplication Tensor
e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal
e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal
Using vertical tensor product, we get
{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}
are 2-orthogonal.
Harm Derksen Tensors, LRMC and Singular Values
![Page 49: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/49.jpg)
Example: Matrix Multiplication Tensor
Theorem (D.)
The matrix multiplication tensor
Tn =n∑
i ,j ,k=1
ei ,j ⊗ ej ,k ⊗ ek,i
is a DSVD.
The singular values of Tn are
1, 1, . . . , 1︸ ︷︷ ︸n3
In particular,
‖Tn‖? =n3∑i=1
1 = n3.
Harm Derksen Tensors, LRMC and Singular Values
![Page 50: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/50.jpg)
Example: Discrete Fourier Transform
DefineFn =
∑1≤i,j,k≤n
i+j+k≡0 mod n
ei ⊗ ej ⊗ ek
This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.
Discrete Fourier Transform (DFT):
Fn =n∑
j=1
√n( 1√
n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei ).
where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are
√n, . . . ,
√n (n times), rank(Fn) = n and ‖Fn‖? = n
√n.
Harm Derksen Tensors, LRMC and Singular Values
![Page 51: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/51.jpg)
Example: Discrete Fourier Transform
DefineFn =
∑1≤i,j,k≤n
i+j+k≡0 mod n
ei ⊗ ej ⊗ ek
This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.
Discrete Fourier Transform (DFT):
Fn =n∑
j=1
√n( 1√
n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei )⊗ ( 1√n
∑ni=1 ζ
ijei ).
where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are
√n, . . . ,
√n (n times), rank(Fn) = n and ‖Fn‖? = n
√n.
Harm Derksen Tensors, LRMC and Singular Values
![Page 52: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/52.jpg)
Generalization: Group Algebra Multiplication Tensor
G is a group with n elements and CG ∼= Cn is the group algebra
TG =∑
g ,h∈Gg ⊗ h ⊗ h−1g−1.
DFT case corresponds to G = Z/nZ.
Theorem (D.)
TG has a DSVD and its singular values are√nd1, . . . ,
√nd1︸ ︷︷ ︸
d31
, . . . ,√
nds, . . . ,
√nds︸ ︷︷ ︸
d3s
where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .
Harm Derksen Tensors, LRMC and Singular Values
![Page 53: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/53.jpg)
Generalization: Group Algebra Multiplication Tensor
G is a group with n elements and CG ∼= Cn is the group algebra
TG =∑
g ,h∈Gg ⊗ h ⊗ h−1g−1.
DFT case corresponds to G = Z/nZ.
Theorem (D.)
TG has a DSVD and its singular values are√nd1, . . . ,
√nd1︸ ︷︷ ︸
d31
, . . . ,√
nds, . . . ,
√nds︸ ︷︷ ︸
d3s
where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .
Harm Derksen Tensors, LRMC and Singular Values
![Page 54: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/54.jpg)
Pn no DSVD
Suppose that Pn has singular values λ1, . . . , λr .
‖Pn‖? = nn/2 =∑r
i=1 λi
‖Pn‖σ = n!nn/2
= λ1
‖Pn‖22 = n! =∑r
i=1 λ2i
λ1
r∑i=1
λi = n! =r∑
i=1
λ2i
so λ1 = · · · = λr , and r = rλ1λ1
= ‖Dn‖?‖Dn‖σ = nn
n! .If n > 2 then r is not an integer!
Harm Derksen Tensors, LRMC and Singular Values
![Page 55: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/55.jpg)
Dn no DSVD
Suppose that Dn has singular values λ1, . . . , λr . Thenλ1 = · · · = λr = 1 and r = n! (similar calculation).
v1, v2, . . . , vr
are 2-orthogonal, so r ≤ (dimV )1/2 = nn/2. So
n! = r ≤ nn/2
Not possible for n > 2.
Harm Derksen Tensors, LRMC and Singular Values
![Page 56: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/56.jpg)
Slope decomposition
Definition
T = T1 + T2 + · · ·+ Ts is called a slope decomposition if〈Ti ,Tj〉 = ‖Ti‖?‖Tj‖σ for all i ≤ j and
‖T1‖?‖T1‖σ
<‖T2‖?‖T2‖σ
< · · · < ‖Tr‖?‖Tr‖σ
.
Slope decomposition is unique if it exists. If T has a DSVD, thenit also has a slope decomposition. But not every tensor has a slopedecomposition.
Harm Derksen Tensors, LRMC and Singular Values
![Page 57: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/57.jpg)
Singular values for tensors with slope decomposition
Definition
Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has
singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −
‖Ti−1‖?‖Ti−1‖σ .
Dn has singular value 1 with multiplicity n!Pn has singular value n!
nn/2with multiplicity nn
n! .
Lemma
If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =
∑i miλi and
‖T‖22 =∑
i miλ2i .
Harm Derksen Tensors, LRMC and Singular Values
![Page 58: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/58.jpg)
Singular values for tensors with slope decomposition
Definition
Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has
singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −
‖Ti−1‖?‖Ti−1‖σ .
Dn has singular value 1 with multiplicity n!Pn has singular value n!
nn/2with multiplicity nn
n! .
Lemma
If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =
∑i miλi and
‖T‖22 =∑
i miλ2i .
Harm Derksen Tensors, LRMC and Singular Values
![Page 59: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/59.jpg)
Singular values for tensors with slope decomposition
Definition
Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has
singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −
‖Ti−1‖?‖Ti−1‖σ .
Dn has singular value 1 with multiplicity n!Pn has singular value n!
nn/2with multiplicity nn
n! .
Lemma
If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =
∑i miλi and
‖T‖22 =∑
i miλ2i .
Harm Derksen Tensors, LRMC and Singular Values
![Page 60: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/60.jpg)
Generalizations
One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.
One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.
Thank You!
Harm Derksen Tensors, LRMC and Singular Values
![Page 61: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/61.jpg)
Generalizations
One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.
One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.
Thank You!
Harm Derksen Tensors, LRMC and Singular Values
![Page 62: Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular](https://reader030.vdocuments.us/reader030/viewer/2022040405/5e9588558ce8b00b927066da/html5/thumbnails/62.jpg)
Generalizations
One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.
One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.
Thank You!
Harm Derksen Tensors, LRMC and Singular Values