![Page 1: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/1.jpg)
Manifold learning, the heat
equation and spectral clustering
Mikhail BelkinDept. of Computer Science and Engineering,
Dept. of Statistics
Ohio State University
Collaborators: Partha Niyogi, Hariharan Narayanan, JianSun, Yusu Wang, Xueyuan Zhou
![Page 2: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/2.jpg)
Ubiquity of manifolds
� In many domains data explicitly lies on a manifold.
� For all sources of high-dimensional data, true dimensionality is much lower than the number of features.
� Much of the data is highly nonlinear.
� In high dimension can trust local but not global distances.
Manifolds (Riemannian manifolds with a measure + noise) provide a natural mathematical language for thinking about high-dimensional data.
![Page 3: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/3.jpg)
From data to graphs to manifolds
How to extract manifold structure from data? Construct a graph to “represent” the underlying space. Properties of the graph should reflect properties of the manifold.
![Page 4: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/4.jpg)
Learning on manifolds
Simple intuition:
Good (plausible) classification function.
Bad (implausible) classification function.
Good functions have low “checkerboardedness”.
![Page 5: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/5.jpg)
How to measure?
But remember we are dealing with data.
Easy enough: graph Laplacian.
![Page 6: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/6.jpg)
Smoothness on manifolds
(Note: this condition not quite strong enough for regularization, but close – more later.)
![Page 7: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/7.jpg)
Laplace operator
Fundamental mathematical object. Heat, wave, Schroedingerequations. Fourier Analysis.
What is Laplace operator on a circle?
![Page 8: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/8.jpg)
Laplace-Beltrami operator
![Page 9: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/9.jpg)
Laplace-Beltrami operator
Nice math, but how to compute from data? Answer: the heat equation.
![Page 10: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/10.jpg)
Laplace operator
![Page 11: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/11.jpg)
Laplace-Beltrami operator
The heat equation has a similar form on manifolds. However do not know distances and the heat kernel.
Turns out (by careful analysis using differential geometry) thatthese issues do not affect algorithms.
![Page 12: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/12.jpg)
Algorithm
![Page 13: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/13.jpg)
Algorithm
Reconstructing eigenfunctions of Laplace-Beltrami operator from sampled data (Laplacian Eigenmaps, Belkin, Niyogi 2001).
1. Construct a data-dependent weighted graph.
2. Compute the bottom eigenvectors of the Laplacian matrix.
Theoretical guarantees as data goes to infinite (using data-dependent t).
Out-of-sample extension?
![Page 14: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/14.jpg)
Non-probabilistic data, such as meshes.
Mesh K, triangle t.
Belkin, Sun, Wang 07,09
Mesh Laplacians
Surface S. Mesh Kε .
Theorem:
Can be extended to arbitrary point clouds. Idea – only need a mesh locally.
![Page 15: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/15.jpg)
Constructing data-dependent bases
Some applications:
� Data representation/visualization.
� Semi-supervised learning.
� Isometry-invariant representations.
Symmetry detection.Ovsyannikov, Sun, Guibas, 2009
![Page 16: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/16.jpg)
16
• In SSL, the more unlabeled data, the better results
we expect
• However... Less unlabeled data More unlabeled data
“Indicator” functions of labeled points.
The more unlabeled data we have, the less stable the classifier gets, and the worse the results become. Laplacian is not powerful enough (has to do with properties of Sobolevspaces). However can use iterated Laplacian. (Recent work with X. Zhou)
Caution about using using Laplacian for regularization
![Page 17: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/17.jpg)
Connections: Locally Linear Embeddings (Roweis, Saul, 2000)
![Page 18: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/18.jpg)
Connections: Locally Linear Embeddings (Roweis, Saul, 2000)
Convergence to the Laplacian in the “limit”. (But the algorithm does not actually
allow that).
![Page 19: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/19.jpg)
Connections: Diffusion Maps (Coifman, Lafon, 2005)
![Page 20: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/20.jpg)
Spectral clustering
![Page 21: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/21.jpg)
Spectral clustering
![Page 22: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/22.jpg)
Spectral clustering : continuous view
![Page 23: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/23.jpg)
Spectral clustering and volumes of cuts
![Page 24: Manifold learning, the heat equation and spectral clustering9.520/spring11/slides/class20_misha.pdf · Mikhail Belkin Dept. of Computer Science and Engineering, ... In many domains](https://reader034.vdocuments.us/reader034/viewer/2022051802/5afa97197f8b9ae92b8df6b5/html5/thumbnails/24.jpg)
Final remarks
Laplacian and the heat equation are a key bridge between data, algorithms and classical mathematics.
Data analysis --- Differential Geometry --- Differential Equations ---Functional Analysis --- Numerical methods --- Algorithms