tensor’dataanalysis - georgia institute of...

Tensor data analysis

Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012

Mariya Ishteva

Scalars

2

Vectors

3

Matrices

4

Tensors

5

Some clarificaFons

!   Nth order tensor

!   DefiniFon !   An element of the tensor product of N vector spaces

!  When the choice of basis is implicit we think of a tensor as its representaFon as an N-‐way array

!   Difficult to visualize !  We will talk mainly about 3rd order tensors !   Results are extendable to higher orders

!   NotaFon !   Not standardized

6

ApplicaFons !   ProbabiliFes

!   matrix: joint/cond. probability table of two variables ,

!   tensor: joint/cond. probability table of a set of variables

!   Text mining: !   matrix: document – term

!   tensor: document – term – year – author

!   Social networks: !   matrix: find communiFes

!   tensor: monitor the change of the community over Fme

!   CollaboraFve filtering !   matrix: user – item

!   tensor: user – item – Fme

!   Signal processing: Example 1 & 3

!   Chemometrics: Example 2

!   Etc. 7

Example 1: EpilepFc seizure onset localizaFon

!   Electrodes

8

!   Electroencephalogram

EpilepFc seizure onset localizaFon: CP model

9

!   CP model / Canonical decomposiFon / Parafac

Example 2: Chemometrics

!   [R. Bro, KVL, Denmark] 10

Fluorescence spectroscopy

!   Demo: !   R. Bro, KVL, Denmark !   hbp://www.models.kvl.dk

11

Example 3: Parameter esFmaFon

12 !   [More details later]

Outline

!   MoFvaFon !   Basic concepts

!   Basic tensor decomposiFons

Next lecture:

!   Other useful decomposiFons !   Local minima

!   Tensors and graphical models

13

Outline


!   Rank and mulFlinear rank

!   Matrix representaFons !   Tensor – matrix mulFplicaFon


14

Matrix rank

!   # linearly independent rows

!   # linearly independent columns !   # rank-‐1 terms

!   Singular value decomposiFon (SVD)

15

Tensor ranks

16

Matrix representaFons of a tensor

!   mulFlinear rank: (rank(A(1)), rank(A(2)), rank(A(3))) 17

Tensor-‐matrix mulFplicaFon

18

Useful matrix operaFons

!   Kronecker product

!   Khatri-‐Rao product !   Column-‐wise Kronecker product

!   Let

19

Scalar product, Frobenius norm, contracFon

!   Scalar product

!   Frobenius norm

!   ContracFon

20

4th order tensor

Outline


!   Basic tensor decomposiFons !   CP / CANDECOMP / PARAFAC

!   MulFlinear SVD

21

Basic decomposiFons

22

CP / Canonical decomposiFon / PARAFAC

23

!   Polyadic form (Hitchcock, 1927)

!   CANDECOMP = Canonical DecomposiFon (Carroll & Chang, 1970) !   PARAFAC = Parallel Factors (Harshman, 1970)

!   Vectors are not necessarily orthogonal

CP: uniqueness

24

!   Kruskal rank k(A): max k, s.t. any k columns are linearly independent

!   Uniqueness

!   Up to permutaFon of the terms

!   Up to scaling of the factors

!   Sufficient condiFon:

!   Note: matrix factorizaFons are not unique

€

A = [a1 a2 a R]B = [b1 b2 b R]C = [c1 c2 c R]

ProperFes of tensor rank !   Not bounded by the dimensions of the tensor

!   CompuFng R: NP-‐hard problem

!   Maximum rank, typical rank

!   Best rank approximaFon: ill-‐posed problem

!   Rank over ≤ rank over !   Example

25

T. Kolda & B. Bader, Tensor decomposiFons and applicaFons SIAM Review, V. 51, # 3, 2009

CompuFng CP

!   Many algorithms

!   AlternaFng least squares: !   Repeat unFl convergence:

!   OpFmize A

!   OpFmize B !   OpFmize C

26

[Example 1]: EpilepFc seizure onset localizaFon

27

!   CP model / Canonical decomposiFon / Parafac

[Example 2]: Chemometrics

!   [R. Bro, KVL, Denmark] 28

Outline


!   Basic tensor decomposiFons !   CP / CANDECOMP / PARAFAC

!   MulFlinear SVD

29

MLSVD decomposiFon

!   3MFA/Tucker3 = Three-‐mode factor analysis (Tucker, 1966)

!   MLSVD = MulFlinear SVD (De Lathauwer, 2000)

!   normalized Tucker decomposiFon

!   U(n): orthogonal !   All-‐orthogonality:

!   Ordering:

30

MLSVD decomposiFon

!   Not unique

!   ComputaFon

!   SVDs of the matrix representaFons A(1), A(2), A(3) U(1),U(2),U(3)

!   A, U(1),U(2),U(3) S

31

€

Α = S •1U(1) •2U

(2) •3U(3)

= (S •1 X−1 •2 Y

−1 •3 Z−1) •1U

(1)X •2U(2)Y •3U

(3)Z

€

S = A •1 (U(1))T •2 (U

(2))T •3 (U(3))T

Low mulFlinear rank approximaFon

32

Low mulFlinear rank approximaFon

33

Higher-‐order orthogonal iteraFon

34

Best rank-‐(R1,R2,R3) approx.: applicaFons

!   ApplicaFon areas !   Chemometrics !   Biomedical signal processing

!   TelecommunicaFons

35

!   Tool for !   Dimensionality reducFon !   Signal subspace esFmaFon

Dimensionality reducFon

36

[Example 3]: Parameter esFmaFon

37


!   HO-‐HTLSstack algorithm

38


!   Tensors can be ill-‐condiFoned in one mode but well-‐condiFoned in other modes. Not possible in matrix case

39

Matlab toolboxes

!   Tensor Toolbox !   hbp://csmr.ca.sandia.gov/~tgkolda/TensorToolbox/ !   B. Bader, T. Kolda and others

!   N-‐way toolbox !   hbp://www.models.life.ku.dk/nwaytoolbox !   R. Bro and C. Andersson

40

Outline



Next lecture:

!   Other useful decomposiFons !   Local minima

!   Tensors and graphical models

41

tensor’dataanalysis - georgia institute of...

Documents