Spectral graph theory

An applied introduction In Lectures on Structural Pattern Recognition @ PRIP

Ernst Schwartz 05/07/2008

Overview   Theory 1: Adjacency, Laplacian,

Eigendecomposition   Applications 1: Embeddings, object retrieval

  Theory 2: Normalized Laplacian   Applications 2: Segmentation, Manifold learning

  Theory 3: Random Walks   Applications 3: Manifold learning, PageRank

  Conclusions

Theory 1: Adjacency, Laplacian, Graph spectrum

 Many ways to represent a graph  Many ways to describe a graph (number of

edges, degree, regularity, ...)  How to gather the most discriminative

information?

Basic notions about graphs   Pairwise affinity

some distance measure for weighting the edges of the graph based on an embedding of the nodes.

For example   Node volume

  Volume of a cluster

  Cut between 2 clusters

Adjacency matrix

G is a simple graph, meaning there are no self edges (E) and at most one edge between two nodes (V)

The corresponding (weighted) adjacency matrix is defined as

Laplacian matrix

with D as the diagonal matrix with D(i,i) = di:

Where di is equal to the degree of the node (eg. the number of edges at that node)

if all weights are 1 (eg. the graph is unweighted) or equal to the volume of the node as seen before

Laplacian matrix

 Describes gradient on the graph

 Strong connection to differential geometry: discrete version of the Laplace-Beltrami operator on manifolds

A (very) simple example

# Adjacency

Laplacian

1 2 3 4 5 6 7 8 9 10 11 12 1 0 1 0 0 0 0 0 0 0 0 0 0 2 1 0 1 0 0 0 0 0 0 0 0 0 3 0 1 0 1 0 0 0 0 0 0 0 0 4 0 0 1 0 1 0 0 0 0 0 0 0 5 0 0 0 1 0 1 0 0 0 0 0 0 6 0 0 0 0 1 0 1 0 0 0 0 0 7 0 0 0 0 0 1 0 1 0 0 0 0 8 0 0 0 0 0 0 1 0 1 0 0 0 9 0 0 0 0 0 0 0 1 0 1 0 0 10 0 0 0 0 0 0 0 0 1 0 1 0 11 0 0 0 0 0 0 0 0 0 1 0 1 12 0 0 0 0 0 0 0 0 0 0 1 0

1 2 3 4 5 6 7 8 9 10 11 12 1 1 -1 0 0 0 0 0 0 0 0 0 0 2 -1 2 -1 0 0 0 0 0 0 0 0 0 3 0 -1 2 -1 0 0 0 0 0 0 0 0 4 0 0 -1 2 -1 0 0 0 0 0 0 0 5 0 0 0 -1 2 -1 0 0 0 0 0 0 6 0 0 0 0 -1 2 -1 0 0 0 0 0 7 0 0 0 0 0 -1 2 -1 0 0 0 0 8 0 0 0 0 0 0 -1 2 -1 0 0 0 9 0 0 0 0 0 0 0 -1 2 -1 0 0 10 0 0 0 0 0 0 0 0 -1 2 -1 0 11 0 0 0 0 0 0 0 0 0 -1 2 -1 12 0 0 0 0 0 0 0 0 0 0 -1 1

(vertices are numbered from 1 to 12)

Eigendecomposition

Computing eigenvalues and eigenvectors of a matrix.

Corresponds to factorizing the matrix into a canonical form.

scalars (eigenvalues)

(eigen)vectors

The following is imposed on the eigenvectors:

A (very) simple example

First four and the last

(eg. ordered by their corresponding eigenvalues)

eigenvector (values of every dimension)

(no vertex information...)

Recap 1

Laplacian strongly connected to gradient

Continuous analogue: Laplace-Beltrami (heat) operator

Application: spectral embedding

 The eigenvectors of the laplacian can be used to compute a (allmost) planar embedding of a mesh.

 No information on the vertices is needed, the embedding is based solely on the connectivity of the mesh.

Original mesh

gplot(A, V)

Spectral embedding

L = diag(sum(A)) - A; [v d] = eigs(L); gpolt(A, v(:,[1 2]));

Application: Object retrieval

  (Deduce mesh from binary image: shock graphs)

 Use spectral information for similarity measures between datasets

 Usefull for image retrieval, classification, ...

Object retrieval

© Jain & Zhang

Application: Brain surface flattening

© Angenent & Haker

Theory 2: Normalized Laplacian

  Laplacian is more informative than the adjacency matrix

 Sometimes cumbersome calculations

 Solution: normalized laplacian !

 Usefull mathematical properties ...

Definition

Properties of the normalized laplacian

Cheegers inequality

Conductance of a vertex set:

Volume of a set of edges joining two subsets:

Conductance of a graph:

Recap 2

  is a very expressive value

 Approximation using Cheeger‘s inequality

 Eigenvalues contain global information about the mesh

Application: Image segmentation

 Generate a graph representation from an image (see last week‘s lecture)

 Find „best“ cuts in that graph

 Spectral interpretation of a „good“ cut: normalized cuts.

Normalized cuts uses the quantity

called the „Normalized Cut“

Normalized Cuts

Define the volume of a set of vertices S as

Normalized Cuts

 Compute eigendecomposition of Laplacian

 Set V = [v1, v2, v3 ... , vk]

 View each row of V as a point in Rk ; cluster into k clusters using k-means ( )

Normalized Cuts

Normalized Cuts

Normalized Cuts

Normalized Cuts

Application: Manifold learning

 New framework for dimensionality reduction  Based on the assumption of an underlying

manifold  Model this manifold implicitly using graph

techniques  Use spectral methods to measure the quality

of the embedding

Manifold learning

?

Manifold learning

?

Manifold learning

?

(semi-supervised learning)

Manifold learning

 What is needed is a way to compute the (geodesic) distance on the data

 For this, we need to be able to describe the structure of the manifold

 Use eigendecomposition of the laplacian !

 Represents geodesic distance !

Manifold learning

How to measure distances ? „swiss roll“ dataset With vertices in 3 dimensions

Manifold learning

Build graph based on k-nns !

Manifold learning

Use spectral embedding using first eigenvectors

First eigenvector

Sec

ond

eige

nvec

tor

Theory 3: Random Walks on graphs

 Use graph structure as a model for a statistical process

 Vertices are states in the system  Edges are transition probabilities for going

from one state to another  Only a small change to the Laplacian

formulation  Good statistical tools available

Definitions

Walk matrix

Taking one step in the walk

Normalized walk matrix

Interesting values of a random walk

 Access/hitting time: expected number of steps before node j is visited

 Commute time expected number of steps in a random walk starting at i, going to j and then back to i

 Cover time expected number of steps to reach every node

 Mixing rate how fast the random walk converges to it‘s limiting distribution

 Access time:

 Commute time:

 Mixing rate:

Spectral computation of graph properties

Recap

 Random walks are a tool for statistical modelling

 Properties can easily be deduced from eigendecomposition

Application: Manifold Learning 2

 Concept: Diffusion on a graph  Propagate a state according to the

connectivity  Discrete time steps equivalent to multiple

resolution analysis  More resistant to noise  Usefull for dimensionality reduction,

segmentation, classification, ...

Diffusion maps

© Coifman & Lafon

Idea: map graph distances to euclidean distances

Use random walks for noise robustness

Very general framework; can be interpreted as

Stochastic differential equations, Graph theoretical methods Physical processes (heat kernel) ...

Connection to kernel methods (!)

Diffusion maps   Based on random walk   Definition of a diffusion distance   Describes distance between points in terms of their

connectivity   Use eigendecomposition to find embedding in euclidean

space (= diffusion maps)   Probability to be in state x at time k can be approximated

using eigendecomposition   More robust to noise due because more robust to

outliers   Extensions: diffusion wavelets using different kernel

sizes

PageRank

© Page & Brim

Recap: Operators on graphs

Operator Spectrum

Adjacency: A

Laplacian: D - A

Normalized Laplacian: I - D-1/2AD-1/2 Random Walk: D-1A

Note: why consider the adjacency as an operator? A1 => nodes reachable from another in one step An => nodes reachable from another in n steps => can be interpreted as an operator on the graph

Conclusions   Tools for graph processing: laplacian & it‘s

eigendecomposition, random walks, diffusions

  All data is structured (even if we don‘t know how)!   Use graphs to represent the underlying structure   Use spectral analysis to find modes of the data.

  Note: classical uses in network analysis... not covered here

Bibliography   Videolectures.net (!!)

  http://videolectures.net/mlcued08_azran_mcl/ Spectral clustering (50min)

  http://videolectures.net/sicgt07_belkin_gmagod/ Graph methods and geometry of data (55min)

  http://videolectures.net/gbr07_hancock_gsis/ Spectral image smoothing (27min)

  http://videolectures.net/mlss05us_nadler_dmscr/ Diffusion maps (55min)

  http://videolectures.net/aop05_hancock_pagt/ Pattern analysis with graphs and trees (120min)

Bibliography   Fan R. Chung; Spectral Graph Theory

(chapters 1-4 online at http://www.math.ucsd.edu/~fan/research/revised.html )

  Daniel Spielman; Spectral Graph Theory and its Applications (online with Matlab code at http://www.cs.yale.edu/homes/spielman/sgta/)

  L. Lovász; Random Walks on Graphs; A Survey   Mauro Maggioni; diffusion maps (matlab code)

http://www.math.duke.edu/~mauro/

Bibliography   Angenent & Haker; On the Laplacian-Beltrami

Operator for Brain Surface flattening   Shi & Malik; Normalized Cuts and Image

Segmentation   Jain & Zhang; A Spectral Approach to Shape-Based

Retrieval of Articulated 3D Models   Cayton; Algorithms for Manifold Learning   Coifmann & Lafon; Diffusion Maps   Page & Brim; The PageRank Citation Ranking:

Bringing Order to the Web