1 unc, stat & or pca extensions for data on manifolds fletcher (principal geodesic anal.) best...

1

UNC, Stat & OR

PCA Extensions for Data on Manifolds

• Fletcher (Principal Geodesic Anal.)• Best fit of geodesic to data

• Constrained to go through geodesic mean

• Huckemann, Hotz & Munk (Geod. PCA)• Best fit of any geodesic to data

• Jung, Foskey & Marron (Princ. Arc Anal.)• Best fit of any circle to data

(motivated by conformal maps)

2

UNC, Stat & OR

PCA Extensions for Data on Manifolds

3

UNC, Stat & OR

Landmark Based Shape Analysis

Key Step: mod out

• Translation

• Scaling

• Rotation

Result:

Data Objects

points on Manifold ( ~ S2k-

4)

4

UNC, Stat & OR

Principal Nested Spheres Analysis

Main Goal:

Extend Principal Arc Analysis (S2 to Sk)

Jung, Dryden & Marron (2012)

5

UNC, Stat & OR


Top Down Nested (small) spheres

6

UNC, Stat & OR


Main Goal:

Extend Principal Arc Analysis (S2 to Sk)

Jung, Dryden & Marron (2012)

Important Landmark: This Motivated

Backwards PCA

7

UNC, Stat & OR


Replace usual forwards view of PCA

Data PC1 (1-d approx)

PC2 (1-d approx of Data-PC1)

PC1 U PC2 (2-d approx)

PC1 U … U PCr

(r-d approx)

8

UNC, Stat & OR


With a backwards approach to PCA

Data PC1 U … U PCr (r-d approx)

PC1 U … U PC(r-1)

PC1 U PC2 (2-d approx)

PC1 (1-d approx)

9

UNC, Stat & OR

Principal Component Analysis

Euclidean Settings:

Forwards PCA = Backwards PCA

(Pythagorean Theorem,

ANOVA Decomposition)

So Not Interesting

But Very Different in Non-Euclidean Settings

(Backwards is Better !?!)

10

UNC, Stat & OR

Principal Component Analysis

Important Property of PCA:

Nested Series of Approximations

(Often taken for granted)

(Desirable in Non-Euclidean Settings)

11

UNC, Stat & OR

How generally applicable is

Backwards approach to PCA?

Where is this already being done???

An Interesting Question

12

UNC, Stat & OR



An Application:

Nonnegative Matrix Factorization

= PCA in Positive Orthant

Think

With ≥ 0 Constraints (on both & )


13

UNC, Stat & OR

Standard Approach:

Lee et al (1999):

Formulate & Solve Optimization

Major Challenge:

Not Nested, ()


14

UNC, Stat & OR

Standard NMF

(Projections

All Inside

Orthant)


15

UNC, Stat & OR

Standard NMF

But Note

Not Nested

No “Multi-scale”

Analysis

Possible (Scores Plot?!?)


16

UNC, Stat & OR

Improved Version:

Use Backwards PCA Idea

“Nonnegative Nested Cone

Analysis”

Collaborator:

Lingsong Zhang (Purdue)

Zhang, Marron, Lu (2013)


17

UNC, Stat & OR

Same Toy

Data Set

All

Projections

In Orthant

Nonnegative Nested Cone Analysis

18

UNC, Stat & OR

Same Toy

Data Set

Rank 1

Approx.

Properly

Nested


19

UNC, Stat & OR

Chemical

Spectral

Data

Gives

Clearer

View


20

UNC, Stat & OR

Chemical

Spectral

Data

Rank 3

Approximation Highlights Lab Early Error


21

UNC, Stat & OR

5-d Toy

Example

(Rainbow

Colored

by Peak

Order)


22

UNC, Stat & OR

5-d Toy Example Rank 1 NNCA Approx.


23

UNC, Stat & OR



24

UNC, Stat & OR


Nonneg.

Basis

Elements

(Not Trivial)


25

UNC, Stat & OR


Current

Research:

How Many

Nonneg.

Basis El’ts

Needed?


26

UNC, Stat & OR



Potential Application: Principal Curves

Hastie & Stuetzle, (1989)

(Foundation of Manifold Learning)


27

UNC, Stat & OR

Goal: Find lower dimensional manifold that well approximates data

ISOmap

Tennenbaum (2000)

Local Linear Embedding

Roweis & Saul (2000)

Manifold Learning

28

UNC, Stat & OR

1st Principal Curve

Linear Reg’n

Usual Smooth

29

UNC, Stat & OR

1st Principal Curve

Linear Reg’n

Proj’s Reg’n

Usual Smooth

30

UNC, Stat & OR

1st Principal Curve

Linear Reg’n

Proj’s Reg’n

Usual Smooth

Princ’l Curve

31

UNC, Stat & OR



Potential Application: Principal Curves

Perceived Major Challenge:

How to find 2nd Principal Curve?

Backwards approach???


32

UNC, Stat & OR

Key Component:

Principal Surfaces

LeBlanc & Tibshirani (1996)


33

UNC, Stat & OR

Key Component:

Principal Surfaces

LeBlanc & Tibshirani (1996)

Challenge:

Can have any dimensional surface,

But how to nest???


34

UNC, Stat & OR



Another Potential Application:

Trees as Data

(early days)


35

UNC, Stat & OR



An Attractive Answer


36

UNC, Stat & OR



An Attractive Answer:

James Damon, UNC Mathematics

Geometry

Singularity

Theory


37

UNC, Stat & OR





Damon and Marron (2013)


38

UNC, Stat & OR





Key Idea: Express Backwards PCA as

Nested Series of Constraints


39

UNC, Stat & OR

Define Nested Spaces via Constraints

Satisfying More Constraints

Smaller Subspaces

General View of Backwards PCA

40

UNC, Stat & OR


E.g. SVD

(Singular Value Decomposition =

= Not Mean Centered PCA)

(notationally very clean)


41

UNC, Stat & OR


E.g. SVD

Have Nested Subspaces:


42

UNC, Stat & OR


E.g. SVD

-th SVD Subspace

Scores

Loading Vectors


43

UNC, Stat & OR


E.g. SVD

Now Define:


44

UNC, Stat & OR


E.g. SVD

Now Define:

Constraint Gives Nested Reduction of Dim’n


45

UNC, Stat & OR


• Backwards PCA

Reduce Using Affine Constraints


46

UNC, Stat & OR


• Backwards PCA

• Principal Nested Spheres

Use Affine Constraints (Planar Slices)

In Ambient Space


47

UNC, Stat & OR


• Backwards PCA


• Principal Surfaces

Spline Constraint Within Previous?


48

UNC, Stat & OR


• Backwards PCA



Spline Constraint Within Previous?

{Been Done Already???}


49

UNC, Stat & OR


• Backwards PCA



• Other Manifold Data Spaces

Sub-Manifold Constraints??

(Algebraic Geometry)


50

UNC, Stat & OR


• Backwards PCA



• Other Manifold Data Spaces

• Tree Spaces

Suitable Constraints???


51

UNC, Stat & OR

New Topic

Curve Registration

52

UNC, Stat & OR

Collaborators

• Anuj Srivastava (Florida State U.)• Wei Wu (Florida State U.)• Derek Tucker (Florida State U.)• Xiaosun Lu (U. N. C.)• Inge Koch (U. Adelaide)• Peter Hoffmann (U. Adelaide)

53

UNC, Stat & OR

Context

Functional Data AnalysisCurves as Data Objects

Toy Example:

54

UNC, Stat & OR

Context


Toy Example:

How Can WeUnderstandVariation?

55

UNC, Stat & OR

Context


Toy Example:


56

UNC, Stat & OR

Context


Toy Example:


57

UNC, Stat & OR

Functional Data Analysis

InsightfulDecomposition

58

UNC, Stat & OR



Horiz’l Var’n

59

UNC, Stat & OR



Vertical Variation

Horiz’l Var’n

60

UNC, Stat & OR

Challenge

Fairly Large Literature

Many (Diverse) Past Attempts

Limited Success (in General)

Surprisingly Slippery

(even mathematical formulation)

61

UNC, Stat & OR

Challenge (Illustrated)

Thanks to Wei Wu

62

UNC, Stat & OR

Challenge (Illustrated)

Thanks to Wei Wu

63

UNC, Stat & OR


AppropriateMathematicalFramework? Vertical Variation

Horiz’l Var’n

64

UNC, Stat & OR


Approach: Identify objects that are:

• Translations

• Rotations

• Scalings

of each other

Mathematics: Equivalence Relation

Results in: Equivalence Classes

Which become the Data Objects

65

UNC, Stat & OR


Equivalence Classes become Data Objects

a.k.a. “Orbits”

Mathematics: Called “Quotient Space”

, , , , , ,

66

UNC, Stat & OR

Curve Registration

What are theData Objects?

Vertical Variation

Horiz’l Var’n

67

UNC, Stat & OR

Curve Registration

What are the Data Objects?

Consider “Time Warpings”

(smooth)

More Precisely: Diffeomorphisms

68

UNC, Stat & OR

Curve Registration

Diffeomorphisms

is 1 to 1 is onto

(thus is invertible) Differentiable is Differentiable

69

UNC, Stat & OR

Time Warping Intuition

Elastically Stretch & Compress Axis

70

UNC, Stat & OR



(identity)

71

UNC, Stat & OR



72

UNC, Stat & OR



73

UNC, Stat & OR



74

UNC, Stat & OR

Curve Registration

Say curves and are equivalent,

When so that

75

UNC, Stat & OR

Curve Registration

Toy Example: Starting Curve,

76

UNC, Stat & OR

Curve Registration

Toy Example: Equivalent Curves,

77

UNC, Stat & OR

Curve Registration

Toy Example: Warping Functions

78

UNC, Stat & OR

Curve Registration

Toy Example: Non-Equivalent Curves

CannotWarpIntoEach Other

79

UNC, Stat & OR

Data Objects I

Equivalence Classes of Curves

(parallel to Kendall shape analysis)

80

UNC, Stat & OR

Data Objects I


(Set of AllWarps ofGiven Curve)

Notation: for a “representor”

81

UNC, Stat & OR

Data Objects I


(Set of AllWarps ofGiven Curve)

Next Task: Find Metric on Equivalence Classes

82

UNC, Stat & OR

Metrics in Curve Space

Find Metric on Equivalence Classes

Start with Warp Invariance on Curves& Extend

83

UNC, Stat & OR


Traditional Approach to Curve

Registration:

• Align curves, say and

• By finding optimal time warp, , so:

• Vertical var’n: PCA after alignment

• Horizontal var’n: PCA on s

84

UNC, Stat & OR


Problem:

Don’t have proper metric

Since:

Because:

85

UNC, Stat & OR


Thanks toXiaosun Lu

86

UNC, Stat & OR


Note:VeryDifferentL2 norms

Thanks toXiaosun Lu

1 unc, stat & or pca extensions for data on manifolds fletcher (principal geodesic anal.) best...

Documents

stat or10how

stat or25how

stat or2landmark

stat or26goal

d approx of data

stat or215d toy example

stat or225d toy example

stat or235d toy example