1

UNC Stat amp OR

DWD in Face Recognition (cont)

Interesting

summary

Jump between

means

(in DWD direction)

Clear separation of

Maleness vs

Femaleness

>

2

UNC Stat amp OR

DWD in Face Recognition (cont)

Fun Comparison

Jump between

means

(in SVM direction)

Also distinguishes

Maleness vs

Femaleness

But not as well as

DWD

>

3

UNC Stat amp OR

DWD in Face Recognition (cont)

Analysis of difference Project onto normals SVM has ldquosmall gaprdquo (feels noise

artifacts) DWD ldquomore informativerdquo (feels real

structure)

HDLSS Discrimrsquon Simulations

Main idea

Comparison of

bull SVM (Support Vector Machine)

bull DWD (Distance Weighted Discrimination)

bull MD (Mean Difference aka Centroid)

Linear versions across dimensions

HDLSS Discrimrsquon Simulations

Overall Approachbull Study different known phenomena

ndash Spherical Gaussiansndash Outliersndash Polynomial Embedding

bull Common Sample Sizes

bull But wide range of dimensions25 nn

16004001004010d

HDLSS Discrimrsquon Simulations

Spherical Gaussians

HDLSS Discrimrsquon Simulations

Spherical Gaussiansbull Same setup as beforebull Means shifted in dim 1 onlybull All methods pretty goodbull Harder problem for higher dimensionbull SVM noticeably worsebull MD best (Likelihood method)bull DWD very close to MDbull Methods converge for higher

dimension

221

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2

UNC Stat amp OR

DWD in Face Recognition (cont)

Fun Comparison

Jump between

means

(in SVM direction)

Also distinguishes

Maleness vs

Femaleness

But not as well as

DWD

>

3

UNC Stat amp OR

DWD in Face Recognition (cont)

Analysis of difference Project onto normals SVM has ldquosmall gaprdquo (feels noise

artifacts) DWD ldquomore informativerdquo (feels real

structure)

HDLSS Discrimrsquon Simulations

Main idea

Comparison of

bull SVM (Support Vector Machine)

bull DWD (Distance Weighted Discrimination)

bull MD (Mean Difference aka Centroid)

Linear versions across dimensions

HDLSS Discrimrsquon Simulations

Overall Approachbull Study different known phenomena

ndash Spherical Gaussiansndash Outliersndash Polynomial Embedding

bull Common Sample Sizes

bull But wide range of dimensions25 nn

16004001004010d

HDLSS Discrimrsquon Simulations

Spherical Gaussians

HDLSS Discrimrsquon Simulations

Spherical Gaussiansbull Same setup as beforebull Means shifted in dim 1 onlybull All methods pretty goodbull Harder problem for higher dimensionbull SVM noticeably worsebull MD best (Likelihood method)bull DWD very close to MDbull Methods converge for higher

dimension

221

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

3

UNC Stat amp OR

DWD in Face Recognition (cont)

Analysis of difference Project onto normals SVM has ldquosmall gaprdquo (feels noise

artifacts) DWD ldquomore informativerdquo (feels real

structure)

HDLSS Discrimrsquon Simulations

Main idea

Comparison of

bull SVM (Support Vector Machine)

bull DWD (Distance Weighted Discrimination)

bull MD (Mean Difference aka Centroid)

Linear versions across dimensions

HDLSS Discrimrsquon Simulations

Overall Approachbull Study different known phenomena

ndash Spherical Gaussiansndash Outliersndash Polynomial Embedding

bull Common Sample Sizes

bull But wide range of dimensions25 nn

16004001004010d

HDLSS Discrimrsquon Simulations

Spherical Gaussians

HDLSS Discrimrsquon Simulations

Spherical Gaussiansbull Same setup as beforebull Means shifted in dim 1 onlybull All methods pretty goodbull Harder problem for higher dimensionbull SVM noticeably worsebull MD best (Likelihood method)bull DWD very close to MDbull Methods converge for higher

dimension

221

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Main idea

Comparison of

bull SVM (Support Vector Machine)

bull DWD (Distance Weighted Discrimination)

bull MD (Mean Difference aka Centroid)

Linear versions across dimensions

HDLSS Discrimrsquon Simulations

Overall Approachbull Study different known phenomena

ndash Spherical Gaussiansndash Outliersndash Polynomial Embedding

bull Common Sample Sizes

bull But wide range of dimensions25 nn

16004001004010d

HDLSS Discrimrsquon Simulations

Spherical Gaussians

HDLSS Discrimrsquon Simulations

Spherical Gaussiansbull Same setup as beforebull Means shifted in dim 1 onlybull All methods pretty goodbull Harder problem for higher dimensionbull SVM noticeably worsebull MD best (Likelihood method)bull DWD very close to MDbull Methods converge for higher

dimension

221

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Overall Approachbull Study different known phenomena

ndash Spherical Gaussiansndash Outliersndash Polynomial Embedding

bull Common Sample Sizes

bull But wide range of dimensions25 nn

16004001004010d

HDLSS Discrimrsquon Simulations

Spherical Gaussians

HDLSS Discrimrsquon Simulations

Spherical Gaussiansbull Same setup as beforebull Means shifted in dim 1 onlybull All methods pretty goodbull Harder problem for higher dimensionbull SVM noticeably worsebull MD best (Likelihood method)bull DWD very close to MDbull Methods converge for higher

dimension

221

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Spherical Gaussians

HDLSS Discrimrsquon Simulations

Spherical Gaussiansbull Same setup as beforebull Means shifted in dim 1 onlybull All methods pretty goodbull Harder problem for higher dimensionbull SVM noticeably worsebull MD best (Likelihood method)bull DWD very close to MDbull Methods converge for higher

dimension

221

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Spherical Gaussiansbull Same setup as beforebull Means shifted in dim 1 onlybull All methods pretty goodbull Harder problem for higher dimensionbull SVM noticeably worsebull MD best (Likelihood method)bull DWD very close to MDbull Methods converge for higher

dimension

221

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Outlier Mixture

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Outlier Mixture80 dim 1 other dims 020 dim 1 plusmn100 dim 2 plusmn500

others 0bull MD is a disaster driven by outliersbull SVM amp DWD are both very robustbull SVM is bestbull DWD very close to SVM (insigrsquot

difference)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Wobble Mixture

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Wobble Mixture80 dim 1 other dims 020 dim 1 plusmn01 rand dim plusmn100

others 0bull MD still very bad driven by outliersbull SVM amp DWD are both very robustbull SVM loses (affected by margin push)bull DWD slightly better (by wrsquoted

influence)bull Methods converge for higher

dimensionIgnore RLR (a mistake)

221

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Nested Spheres

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon SimulationsNested Spheres

1st d2 dimrsquos Gaussian with var 1 or C2nd d2 dimrsquos the squares of the 1st

dimrsquos(as for 2nd degree polynomial

embedding) bull Each method best somewherebull MD best in highest d (data non-

Gaussian)bull Methods not comparable (realistic)bull Methods converge for higher

dimensionbull HDLSS space is a strange place

Ignore RLR (a mistake)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon SimulationsConclusions

bull Everything (sensible) is best sometimes

bull DWD often very near bestbull MD weak beyond Gaussian

Caution about simulations (and examples)

bull Very easy to cherry pick best onesbull Good practice in Machine Learning

ndash ldquoIgnore method proposed but read paper for useful comparison of

othersrdquo

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Caution There are additional players

Eg Regularized Logistic Regression

looks also very competitive

Interesting Phenomenon

All methods come together

in very high dimensions

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

HDLSS Discrimrsquon Simulations

Can we say more about

All methods come together

in very high dimensions

Mathematical Statistical Question

Mathematics behind this

(will answer later)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM amp DWD Tuning Parameter

Main Idea

Handling of Violators (ldquoSlack Variablesrdquo)

Controlled by Tuning Parameter C

Larger C Try Harder to Avoid Violation

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM Tuning ParameterRecall Movie for SVM

>

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

(Can be Effective

But Takes Time

Requires Expertise)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

DWD 100 median pairwise

distance

(Surprisingly Useful Simple Answer)

SVM 1000

(Works Well Sometimes Not Others)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

(Works Well for DWD

Less Effective for SVM)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

Measure Classification Error Rate

Leaving Some Out (to Avoid Overfitting)

Choose C to Minimize Error Rate

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

(Very Popular ndash Useful for SVD

But Comes at Computational Cost)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

SVM amp DWD Tuning Parameter

Possible Approaches

bull Visually Tuned

bull Simple Defaults

bull Cross Validation

bull Scale Space

(Work with Full Range of Choices)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Use Image Features as Before

(Recall from Transformation Discussion)

Paper Miedema et al (2012)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

March 17 2010 26

Clinical diagnosis

BackgroundIntroduction

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

March 17 2010 27

Image Analysis of Histology Slides

GoalBackground

Melanoma

Image wwwmelanomaca

Benign1 in 75 North Americans will develop a malignant melanoma in their lifetime

Initial goal Automatically segment nucleiChallenge Dense packing of nucleiUltimately Cancer grading and patient survival

Image melanomablogsomecom

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

March 17 2010 28

Feature Extraction

Features from Cell NucleiFeature Extraction

Extract various features based on color and morphology

Example ldquohigh-levelrdquo concepts

bull Stain intensity

bull Nuclear area

bull Density of nuclei

bull Regularity of nuclear shape

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

March 17 2010 29

Labeled Nuclei

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

March 17 2010 30

Nuclear Regions

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Generated by growing nuclei out from boundary

Used for various color and density features Region Stain 2 Region Area Ratio etc

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

March 17 2010 31

Delaunay Triangulation

Features from Cell NucleiFeature Extraction

Conventional Nevus Superficial Spreading Melanoma

Triangulation of nuclear centers

Used for various density features Mean Delaunay Max Delaunay etc

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Databull Study Differences Between

(Malignant) Melanoma amp (Benign) Nevi

Explore with PCA View

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

PCA

View

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate

To DWD

Direction

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate

To DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate

To DWD

Direction

Orthogonal

PCs

Avoid

Strange

Projections

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Return

To

PCA

View

And

Focus

On

Subtypes

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Focus

On

Subtypes

Melanoma 1

Sev Dys

Nevi

Gray Out

Others

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate

To

Pairwise

Only

PCA

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate

To DWD

amp Ortho PCs

Better

Separation

Than Full

Data

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Full Data

DWD

Direction

ldquoGoodrdquo

Separation

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Developed in WWII History in

Green and Swets (1966)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Approach from Signal Detection

Receiver Operator Characteristic

(ROC) Curve

Good Modern Treatment

DeLong DeLong amp Clarke-Pearson (1988)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Challenge

Measure ldquoGoodness of Separationrdquo

Idea For Range of Cutoffs

Plot

Proprsquon +1rsquos Smaller than Cutoff

Vs

Proprsquon -1s Smaller than Cutoff

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Aim

Quantify

ldquoOverlaprdquo

Approach

Consider

Series

Of Cutoffs

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Approach

Consider

Series

Of Cutoffs

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

X-coord

Is Proprsquon

Of Reds

Smaller

Y-coord

Is Proprsquon

Of Blues

Smaller

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Slide

Cutoff

To

Trace

Out

Curve

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Better

Separation

Is ldquoMore

To Upper

Leftrdquo

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Summarize

amp Compare

Using

Area

Under

Curve

(AUC)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

Perfect

Separation

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

Very

Slight

Overlap

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

Little

More

Overlap

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

More

Overlap

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

Much

More

Overlap

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

Complete

Overlap

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

Complete

Overlap

AUC asymp 05 Reflects ldquoCoin Tossingrdquo

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Toy

Example

Can

Reflect

ldquoWorse

Than Coin

Tossingrdquo

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ROC Curve

Interpretation of AUC

Very Context Dependent

Radiology

ldquogt 70 has Predictive Usefulnessrdquo

Bigger is Better

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Subclass

DWD

Direction

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Full Data

DWD

Direction

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Recall Question

Which Gives Better Separation of

Melanoma vs Nevi

DWD on All Melanoma vs All Nevi DWD on Melanoma 1 vs Sev Dys Nevi

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Full Data

ROC

Analysis

AUC = 093

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

SubClass

ROC

Analysis

AUC = 095

Better

Makes

Intuitive

Sense

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

What About Other Subclasses

Looked at Several

Best Separation Was

Melanoma 2 vs Conventional Nevi

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Full

Data

PCA

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Full

Data

PCA

Gray

Out

All

But

Subclasses

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate to

SubClass

PCA

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

Rotate to

SubClass

DWD

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

Melanoma Data

ROC

Analysis

AUC = 099

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ClusteringIdea Given data bull Assign each object to a class

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data driven

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

nXX

1

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ClusteringIdea Given data bull Assign each object to a classbull Of similar objectsbull Completely data drivenbull Ie assign labels to databull ldquoUnsupervised Learningrdquo

Contrast to Classification (Discrimination)bull With predetermined classesbull ldquoSupervised Learningrdquo

nXX

1

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

ClusteringImportant Referencesbull MacQueen (1967)bull Hartigan (1975)bull Gersho and Gray (1992)bull Kaufman and Rousseeuw (2005)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringMain Idea for data

Partition indices

among classes

nXX

1

ni 1

KCC 1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition

nXX

1

ni 1

KCC 1

KCC 1

n1

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringMain Idea for data

Partition indices

among classes

Given index sets bull that partition bull represent clusters by ldquoclass meansrdquo

ie (within class means)

nXX

1

ni 1

KCC 1

KCC 1

jCi

ij

j XC

X

1

n1

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringGiven index sets

Measure how well clustered using

Within Class Sum of Squares

KCC 1

2

1

jCi

ji

K

j

XX

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringCommon Variation

Put on scale of proportions (ie in [01])

By dividing ldquowithin class SSrdquo

by ldquooverall SSrdquo

Gives Cluster Index

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster means

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means ClusteringNotes on Cluster Index

bull CI = 0 when all data at cluster meansbull CI small when gives tight clustering

(within SS contains little variation)bull CI big when gives poor clustering

(within SS contains most of variation)bull CI = 1 when all cluster means are same

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

KCC 1

KCC 1

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

K-means Clustering

Clustering Goal

bull Given data

bull Choose classes

bull To miminize

KCC 1

nXX

1

n

ii

Ciji

K

j

K

XX

XX

CCCI j

1

2

2

1

1

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Varying Standard Deviation

bull Varying Mean

bull Varying Proportion

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Multiple local minima (large number)

ndash Maybe disconnected

ndash Optimization (over ) can be trickyhellip

(even in 1 dimension with K = 2)

KCC 1

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Can have 4 (or more) local mins

(even in 1 dimension with K = 2)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Over changing Classes (moving brsquodry)

bull Multi-modal data interesting effects

ndash Local mins can be hard to find

ndash ie iterative procedures can ldquoget stuckrdquo

(even in 1 dimension with K = 2)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)

2-means Clustering

Study CI using simple 1-d examples

bull Effect of a single outlier

ndash Can create local minimum

ndash Can also yield a global minimum

ndash This gives a one point class

ndash Can make CI arbitrarily small

(really a ldquogood clusteringrdquo)

• DWD in Face Recognition (cont)
• DWD in Face Recognition (cont) (2)
• DWD in Face Recognition (cont) (3)
• HDLSS Discrimrsquon Simulations
• HDLSS Discrimrsquon Simulations (2)
• HDLSS Discrimrsquon Simulations (3)
• HDLSS Discrimrsquon Simulations (4)
• HDLSS Discrimrsquon Simulations (5)
• HDLSS Discrimrsquon Simulations (6)
• HDLSS Discrimrsquon Simulations (7)
• HDLSS Discrimrsquon Simulations (8)
• HDLSS Discrimrsquon Simulations (9)
• HDLSS Discrimrsquon Simulations (10)
• HDLSS Discrimrsquon Simulations (11)
• HDLSS Discrimrsquon Simulations (12)
• HDLSS Discrimrsquon Simulations (13)
• SVM amp DWD Tuning Parameter
• SVM Tuning Parameter
• SVM amp DWD Tuning Parameter (2)
• SVM amp DWD Tuning Parameter (3)
• SVM amp DWD Tuning Parameter (4)
• SVM amp DWD Tuning Parameter (5)
• SVM amp DWD Tuning Parameter (6)
• SVM amp DWD Tuning Parameter (7)
• Melanoma Data
• Slide 26
• Slide 27
• Slide 28
• Slide 29
• Slide 30
• Slide 31
• Melanoma Data (2)
• Melanoma Data (3)
• Melanoma Data (4)
• Melanoma Data (5)
• Melanoma Data (6)
• Melanoma Data (7)
• Melanoma Data (8)
• Melanoma Data (9)
• Melanoma Data (10)
• Melanoma Data (11)
• Melanoma Data (12)
• Melanoma Data (13)
• ROC Curve
• ROC Curve (2)
• ROC Curve (3)
• ROC Curve (4)
• ROC Curve (5)
• ROC Curve (6)
• ROC Curve (7)
• ROC Curve (8)
• ROC Curve (9)
• ROC Curve (10)
• ROC Curve (11)
• ROC Curve (12)
• ROC Curve (13)
• ROC Curve (14)
• ROC Curve (15)
• ROC Curve (16)
• ROC Curve (17)
• ROC Curve (18)
• ROC Curve (19)
• ROC Curve (20)
• ROC Curve (21)
• ROC Curve (22)
• ROC Curve (23)
• Melanoma Data (14)
• Melanoma Data (15)
• Melanoma Data (16)
• Melanoma Data (17)
• Melanoma Data (18)
• Melanoma Data (19)
• Melanoma Data (20)
• Melanoma Data (21)
• Melanoma Data (22)
• Melanoma Data (23)
• Melanoma Data (24)
• Melanoma Data (25)
• Clustering
• Clustering (2)
• Clustering (3)
• Clustering (4)
• Clustering (5)
• K-means Clustering
• K-means Clustering (2)
• K-means Clustering (3)
• K-means Clustering (4)
• K-means Clustering (5)
• K-means Clustering (6)
• K-means Clustering (7)
• K-means Clustering (8)
• K-means Clustering (9)
• K-means Clustering (10)
• K-means Clustering (11)
• K-means Clustering (12)
• 2-means Clustering
• 2-means Clustering (2)
• 2-means Clustering (3)
• 2-means Clustering (4)
• 2-means Clustering (5)
• 2-means Clustering (6)
• 2-means Clustering (7)
• 2-means Clustering (8)
• 2-means Clustering (9)
• 2-means Clustering (10)
• 2-means Clustering (11)
• 2-means Clustering (12)
• 2-means Clustering (13)
• 2-means Clustering (14)
• 2-means Clustering (15)
• 2-means Clustering (16)
• 2-means Clustering (17)
• 2-means Clustering (18)
• 2-means Clustering (19)
• 2-means Clustering (20)
• 2-means Clustering (21)
• 2-means Clustering (22)
• 2-means Clustering (23)
• 2-means Clustering (24)
• 2-means Clustering (25)
• 2-means Clustering (26)
• 2-means Clustering (27)
• 2-means Clustering (28)
• 2-means Clustering (29)
• 2-means Clustering (30)
• 2-means Clustering (31)
• 2-means Clustering (32)
• 2-means Clustering (33)
• 2-means Clustering (34)
• 2-means Clustering (35)
• 2-means Clustering (36)
• 2-means Clustering (37)
• 2-means Clustering (38)
• 2-means Clustering (39)
• 2-means Clustering (40)
• 2-means Clustering (41)
• 2-means Clustering (42)
• 2-means Clustering (43)
• 2-means Clustering (44)
• 2-means Clustering (45)
• 2-means Clustering (46)
• 2-means Clustering (47)
• 2-means Clustering (48)
• 2-means Clustering (49)
• 2-means Clustering (50)
• 2-means Clustering (51)
• 2-means Clustering (52)
• 2-means Clustering (53)
• 2-means Clustering (54)
• 2-means Clustering (55)
• 2-means Clustering (56)
• 2-means Clustering (57)
• 2-means Clustering (58)
• 2-means Clustering (59)
• 2-means Clustering (60)
• 2-means Clustering (61)
• 2-means Clustering (62)
• 2-means Clustering (63)
• 2-means Clustering (64)
• 2-means Clustering (65)
• 2-means Clustering (66)
• 2-means Clustering (67)
• 2-means Clustering (68)