1 unc, stat & or pca extensions for data on manifolds fletcher (principal geodesic anal.) best...

1

UNC, Stat & OR

PCA Extensions for Data on Manifolds

• Fletcher (Principal Geodesic Anal.)• Best fit of geodesic to data

• Constrained to go through geodesic mean

• Huckemann, Hotz & Munk (Geod. PCA)• Best fit of any geodesic to data

• Jung, Foskey & Marron (Princ. Arc Anal.)• Best fit of any circle to data

(motivated by conformal maps)

2

UNC, Stat & OR

PCA Extensions for Data on Manifolds

3

UNC, Stat & OR

Landmark Based Shape Analysis

Key Step: mod out

• Translation

• Scaling

• Rotation

Result:

Data Objects

points on Manifold ( ~ S2k-

4)

4

UNC, Stat & OR

Principal Nested Spheres Analysis

Main Goal:

Extend Principal Arc Analysis (S2 to Sk)

Jung, Dryden & Marron (2012)

5

UNC, Stat & OR

Principal Nested Spheres Analysis

Top Down Nested (small) spheres

6

UNC, Stat & OR

Principal Nested Spheres Analysis

Main Goal:

Extend Principal Arc Analysis (S2 to Sk)

Jung, Dryden & Marron (2012)

Important Landmark: This Motivated

Backwards PCA

7

UNC, Stat & OR

Principal Nested Spheres Analysis

Replace usual forwards view of PCA

Data PC1 (1-d approx)

PC2 (1-d approx of Data-PC1)

PC1 U PC2 (2-d approx)

PC1 U … U PCr

(r-d approx)

8

UNC, Stat & OR

Principal Nested Spheres Analysis

With a backwards approach to PCA

Data PC1 U … U PCr (r-d approx)

PC1 U … U PC(r-1)

PC1 U PC2 (2-d approx)

PC1 (1-d approx)

9

UNC, Stat & OR

Principal Component Analysis

Euclidean Settings:

Forwards PCA = Backwards PCA

(Pythagorean Theorem,

ANOVA Decomposition)

So Not Interesting

But Very Different in Non-Euclidean Settings

(Backwards is Better !?!)

10

UNC, Stat & OR

Principal Component Analysis

Important Property of PCA:

Nested Series of Approximations

(Often taken for granted)

(Desirable in Non-Euclidean Settings)

11

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

Where is this already being done???

An Interesting Question

12

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

An Application:

Nonnegative Matrix Factorization

= PCA in Positive Orthant

Think

With ≥ 0 Constraints (on both & )

An Interesting Question

13

UNC, Stat & OR

Standard Approach:

Lee et al (1999):

Formulate & Solve Optimization

Major Challenge:

Not Nested, ()

Nonnegative Matrix Factorization

14

UNC, Stat & OR

Standard NMF

(Projections

All Inside

Orthant)

Nonnegative Matrix Factorization

15

UNC, Stat & OR

Standard NMF

But Note

Not Nested

No “Multi-scale”

Analysis

Possible (Scores Plot?!?)

Nonnegative Matrix Factorization

16

UNC, Stat & OR

Improved Version:

Use Backwards PCA Idea

“Nonnegative Nested Cone

Analysis”

Collaborator:

Lingsong Zhang (Purdue)

Zhang, Marron, Lu (2013)

Nonnegative Matrix Factorization

17

UNC, Stat & OR

Same Toy

Data Set

All

Projections

In Orthant

Nonnegative Nested Cone Analysis

18

UNC, Stat & OR

Same Toy

Data Set

Rank 1

Approx.

Properly

Nested

Nonnegative Nested Cone Analysis

19

UNC, Stat & OR

Chemical

Spectral

Data

Gives

Clearer

View

Nonnegative Nested Cone Analysis

20

UNC, Stat & OR

Chemical

Spectral

Data

Rank 3

Approximation Highlights Lab Early Error

Nonnegative Nested Cone Analysis

21

UNC, Stat & OR

5-d Toy

Example

(Rainbow

Colored

by Peak

Order)

Nonnegative Nested Cone Analysis

22

UNC, Stat & OR

5-d Toy Example Rank 1 NNCA Approx.

Nonnegative Nested Cone Analysis

23

UNC, Stat & OR

5-d Toy Example Rank 2 NNCA Approx.

Nonnegative Nested Cone Analysis

24

UNC, Stat & OR

5-d Toy Example Rank 2 NNCA Approx.

Nonneg.

Basis

Elements

(Not Trivial)

Nonnegative Nested Cone Analysis

25

UNC, Stat & OR

5-d Toy Example Rank 3 NNCA Approx.

Current

Research:

How Many

Nonneg.

Basis El’ts

Needed?

Nonnegative Nested Cone Analysis

26

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

Potential Application: Principal Curves

Hastie & Stuetzle, (1989)

(Foundation of Manifold Learning)

An Interesting Question

27

UNC, Stat & OR

Goal: Find lower dimensional manifold that well approximates data

ISOmap

Tennenbaum (2000)

Local Linear Embedding

Roweis & Saul (2000)

Manifold Learning

28

UNC, Stat & OR

1st Principal Curve

Linear Reg’n

Usual Smooth

29

UNC, Stat & OR

1st Principal Curve

Linear Reg’n

Proj’s Reg’n

Usual Smooth

30

UNC, Stat & OR

1st Principal Curve

Linear Reg’n

Proj’s Reg’n

Usual Smooth

Princ’l Curve

31

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

Potential Application: Principal Curves

Perceived Major Challenge:

How to find 2nd Principal Curve?

Backwards approach???

An Interesting Question

32

UNC, Stat & OR

Key Component:

Principal Surfaces

LeBlanc & Tibshirani (1996)

An Interesting Question

33

UNC, Stat & OR

Key Component:

Principal Surfaces

LeBlanc & Tibshirani (1996)

Challenge:

Can have any dimensional surface,

But how to nest???

An Interesting Question

34

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

Another Potential Application:

Trees as Data

(early days)

An Interesting Question

35

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

An Attractive Answer

An Interesting Question

36

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

An Attractive Answer:

James Damon, UNC Mathematics

Geometry

Singularity

Theory

An Interesting Question

37

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

An Attractive Answer:

James Damon, UNC Mathematics

Damon and Marron (2013)

An Interesting Question

38

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

An Attractive Answer:

James Damon, UNC Mathematics

Key Idea: Express Backwards PCA as

Nested Series of Constraints

An Interesting Question

39

UNC, Stat & OR

Define Nested Spaces via Constraints

Satisfying More Constraints

Smaller Subspaces

General View of Backwards PCA

40

UNC, Stat & OR

Define Nested Spaces via Constraints

E.g. SVD

(Singular Value Decomposition =

= Not Mean Centered PCA)

(notationally very clean)

General View of Backwards PCA

41

UNC, Stat & OR

Define Nested Spaces via Constraints

E.g. SVD

Have Nested Subspaces:

General View of Backwards PCA

42

UNC, Stat & OR

Define Nested Spaces via Constraints

E.g. SVD

-th SVD Subspace

Scores

Loading Vectors

General View of Backwards PCA

43

UNC, Stat & OR

Define Nested Spaces via Constraints

E.g. SVD

Now Define:

General View of Backwards PCA

44

UNC, Stat & OR

Define Nested Spaces via Constraints

E.g. SVD

Now Define:

Constraint Gives Nested Reduction of Dim’n

General View of Backwards PCA

45

UNC, Stat & OR

Define Nested Spaces via Constraints

• Backwards PCA

Reduce Using Affine Constraints

General View of Backwards PCA

46

UNC, Stat & OR

Define Nested Spaces via Constraints

• Backwards PCA

• Principal Nested Spheres

Use Affine Constraints (Planar Slices)

In Ambient Space

General View of Backwards PCA

47

UNC, Stat & OR

Define Nested Spaces via Constraints

• Backwards PCA

• Principal Nested Spheres

• Principal Surfaces

Spline Constraint Within Previous?

General View of Backwards PCA

48

UNC, Stat & OR

Define Nested Spaces via Constraints

• Backwards PCA

• Principal Nested Spheres

• Principal Surfaces

Spline Constraint Within Previous?

{Been Done Already???}

General View of Backwards PCA

49

UNC, Stat & OR

Define Nested Spaces via Constraints

• Backwards PCA

• Principal Nested Spheres

• Principal Surfaces

• Other Manifold Data Spaces

Sub-Manifold Constraints??

(Algebraic Geometry)

General View of Backwards PCA

50

UNC, Stat & OR

Define Nested Spaces via Constraints

• Backwards PCA

• Principal Nested Spheres

• Principal Surfaces

• Other Manifold Data Spaces

• Tree Spaces

Suitable Constraints???

General View of Backwards PCA

51

UNC, Stat & OR

New Topic

Curve Registration

52

UNC, Stat & OR

Collaborators

• Anuj Srivastava (Florida State U.)• Wei Wu (Florida State U.)• Derek Tucker (Florida State U.)• Xiaosun Lu (U. N. C.)• Inge Koch (U. Adelaide)• Peter Hoffmann (U. Adelaide)

53

UNC, Stat & OR

Context

Functional Data AnalysisCurves as Data Objects

Toy Example:

54

UNC, Stat & OR

Context

Functional Data AnalysisCurves as Data Objects

Toy Example:

How Can WeUnderstandVariation?

55

UNC, Stat & OR

Context

Functional Data AnalysisCurves as Data Objects

Toy Example:

How Can WeUnderstandVariation?

56

UNC, Stat & OR

Context

Functional Data AnalysisCurves as Data Objects

Toy Example:

How Can WeUnderstandVariation?

57

UNC, Stat & OR

Functional Data Analysis

InsightfulDecomposition

58

UNC, Stat & OR

Functional Data Analysis

InsightfulDecomposition

Horiz’l Var’n

59

UNC, Stat & OR

Functional Data Analysis

InsightfulDecomposition

Vertical Variation

Horiz’l Var’n

60

UNC, Stat & OR

Challenge

Fairly Large Literature

Many (Diverse) Past Attempts

Limited Success (in General)

Surprisingly Slippery

(even mathematical formulation)

61

UNC, Stat & OR

Challenge (Illustrated)

Thanks to Wei Wu

62

UNC, Stat & OR

Challenge (Illustrated)

Thanks to Wei Wu

63

UNC, Stat & OR

Functional Data Analysis

AppropriateMathematicalFramework? Vertical Variation

Horiz’l Var’n

64

UNC, Stat & OR

Landmark Based Shape Analysis

Approach: Identify objects that are:

• Translations

• Rotations

• Scalings

of each other

Mathematics: Equivalence Relation

Results in: Equivalence Classes

Which become the Data Objects

65

UNC, Stat & OR

Landmark Based Shape Analysis

Equivalence Classes become Data Objects

a.k.a. “Orbits”

Mathematics: Called “Quotient Space”

, , , , , ,

66

UNC, Stat & OR

Curve Registration

What are theData Objects?

Vertical Variation

Horiz’l Var’n

67

UNC, Stat & OR

Curve Registration

What are the Data Objects?

Consider “Time Warpings”

(smooth)

More Precisely: Diffeomorphisms

68

UNC, Stat & OR

Curve Registration

Diffeomorphisms

is 1 to 1 is onto

(thus is invertible) Differentiable is Differentiable

69

UNC, Stat & OR

Time Warping Intuition

Elastically Stretch & Compress Axis

70

UNC, Stat & OR

Time Warping Intuition

Elastically Stretch & Compress Axis

(identity)

71

UNC, Stat & OR

Time Warping Intuition

Elastically Stretch & Compress Axis

72

UNC, Stat & OR

Time Warping Intuition

Elastically Stretch & Compress Axis

73

UNC, Stat & OR

Time Warping Intuition

Elastically Stretch & Compress Axis

74

UNC, Stat & OR

Curve Registration

Say curves and are equivalent,

When so that

75

UNC, Stat & OR

Curve Registration

Toy Example: Starting Curve,

76

UNC, Stat & OR

Curve Registration

Toy Example: Equivalent Curves,

77

UNC, Stat & OR

Curve Registration

Toy Example: Warping Functions

78

UNC, Stat & OR

Curve Registration

Toy Example: Non-Equivalent Curves

CannotWarpIntoEach Other

79

UNC, Stat & OR

Data Objects I

Equivalence Classes of Curves

(parallel to Kendall shape analysis)

80

UNC, Stat & OR

Data Objects I

Equivalence Classes of Curves

(Set of AllWarps ofGiven Curve)

Notation: for a “representor”

81

UNC, Stat & OR

Data Objects I

Equivalence Classes of Curves

(Set of AllWarps ofGiven Curve)

Next Task: Find Metric on Equivalence Classes

82

UNC, Stat & OR

Metrics in Curve Space

Find Metric on Equivalence Classes

Start with Warp Invariance on Curves& Extend

83

UNC, Stat & OR

Metrics in Curve Space

Traditional Approach to Curve

Registration:

• Align curves, say and

• By finding optimal time warp, , so:

• Vertical var’n: PCA after alignment

• Horizontal var’n: PCA on s

84

UNC, Stat & OR

Metrics in Curve Space

Problem:

Don’t have proper metric

Since:

Because:

85

UNC, Stat & OR

Metrics in Curve Space

Thanks toXiaosun Lu

86

UNC, Stat & OR

Metrics in Curve Space

Note:VeryDifferentL2 norms

Thanks toXiaosun Lu