Effective-resistance Preserving Spectral Reduction of Graphs

Zhiqiang Zhao Zhuo Feng

Design Automation Group

Department of Electrical & Computer EngineeringMichigan Technological University

Acknowledgements: NSF CCF grants #1350206, #1318694 & #1618364, KeySight Technologies.

Increasing Complexity of Graphs & Networks A graph or network: nodes and (weighted) edges

Networks are only getting bigger– Number of transistors in integrated circuits doubles every two years

– Data and social networks are evolving every second

How to deal with the fast growing networks?

Figures source: Wikipedia

Social Networks Transportation NetworksCircuit Networks

Spectral Graph Reduction Motivation

– Reduce graph complexity while preserving key spectral properties:• Laplacian eigenvalues / eigenvectors

• Effective resistances

• Low dimensional embedding

• Pair-wise distances

• Cuts between nodes

• …

– Key to development of faster numerical and graph-related algorithms:• Solving large PDEs & sparse matrices, graph partitioning & data clustering, semi-supervised learning

(SSL), graph convolution neural networks, maximum flows of undirected graphs, …

– Allow for efficient handling of big (data) graphs• Big graph computing on personal computers, or even mobile devices

Spectral Reduction

Applications in VLSI CAD Spectrally-reduced circuits preserve effective resistances [1]

– Opportunity: development of much faster VLSI CAD algorithms

2

4 6

8 11

13 14

1 18

1621 17 22

25 27

2

4 6

8 11

13 14

1 18

1621 17 22

25 27

Ci1

VinCs1

clk1 clk2

clk2 clk1

Ci2

Cs2

clk1 clk2

clk2 clk1

y1y2

Cf1

clk2 clk1

clk1 clk2

Cf2

clk2 clk1

clk1 clk2

Comparator

Q

_Q

D flip-flop

CLK1

d

d1 d2

0 50 100 150 2000

50

100

150

200

2

4

6

8

10

12

x 10-3

Spectral Graph Approximation

Spectral Circuit Sparsification & Reduction

Nearly-Linear Time VLSI CAD Algorithms

M2M1

R7

M5

L1

L0C0

Vlo+ M3 M4

M6

R1

R3

R8

L2

R10

L3C1

R2

Vrf+ R5 Vrf-R6

Vlo-R4

VDD

[1] [8]

[21 ] [16]

[25] [27]

[20] [7]

[15]

[26 ]

[13] [14][11] [18]

[22][17]

[4] [6]

[2]

[21]

[2]

[1] [8]

[16]

[25] [27]

[20] [7]

[15]

[13] [14][11] [18]

[22][17]

[4] [6]

[1] Z. Zhao, and Z. Feng. A spectral graph sparsification approach to scalable vectorless power grid integrity verification. DAC’17.

reduced graph original graph reduced graph

Applications in Machine Learning & Data Mining Graph Convolutional Neural Networks deal with unstructured data [1]

– Numerous applications: graph embedding [3], semi-supervised learning [2],…

– Spectrally-reduced graphs preserve key features & reduce computation cost

[1] M. Defferrard, et al. Convolutional neural networks on graphs with fast localized spectral filtering. NIPS’16[2] T. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ICLR’17[3] Hamilton, Will, et al. "Inductive representation learning on large graphs." In NIPS’17

Figure modified from [1]

Overview of Our Approach

Graph Density?

Spectral Graph Reduction

Original Graph

C. Effective-ResistancePreserving Post Scaling

Spectrally Reduced Graph

B. Spectral Graph Sparsification & Scaling

A. Spectrum-Preserving Node Reduction

High

B. Spectral Graph Sparsification & Scaling

A. Spectrum-Preserving Node Reduction

Low

Original Graph

Reduced Graph

Sparsified Reduced Graph

A

B & C

Graph Laplacian Matrix Graph Laplacian matrix (admittance matrix of a resistor network)

– Graph 𝐺𝐺 = 𝑉𝑉,𝐸𝐸,𝑤𝑤 and its Laplacian matrix 𝐿𝐿𝐺𝐺:• 𝑳𝑳𝑮𝑮 is symmetric diagonally dominant (SDD)• Eigenvalues of 𝑳𝑳𝑮𝑮 are nonnegative

( , )

( , ) if ( , )( , ) ( , ) if

otherwise0G u v E

w u v u v EL u v w u v u v

∈

− ∈

= ==

∑

1 2

4

3

5

1.5

22

1.51

0.5

A Weighted Graph 𝑮𝑮

3.5 1.5 21.5 4 2 0.5

2 3 10.5 1 3 1.5

2 1.5 3.5

− − − − − − − − − − − −

Graph Laplacian 𝑳𝑳𝑮𝑮

Laplacian eigenvectors vs signals in Fourier analysis– Higher “frequencies” in Fourier analysis higher eigenvalues [1]

– “Low-frequency” eigenvectors graph global properties

– Our goal: retain “low-frequency” eigenvectors w/ fewest nodes

𝝎𝝎𝟑𝟑

Set 1Set 2

Set 3

𝜁𝜁3 > 𝜁𝜁2

Laplacian Eigenvectors as Signals on Graphs

[1] D. Shuman, et al. "The emerging field of signal processing on graphs: extending high-dimensional data analysis to networks and other irregular domains." IEEE Signal Processing Magazine (2013)

𝝎𝝎𝟐𝟐Set 1 Set 2

“Res

onan

t Fre

quen

cy”

𝜁𝜁2 > 𝜁𝜁1=0

𝐿𝐿𝐺𝐺 = �𝑖𝑖=1

𝑛𝑛

𝜁𝜁𝑖𝑖𝜔𝜔𝑖𝑖𝜔𝜔𝑖𝑖𝑇𝑇 , 𝜁𝜁𝑖𝑖 : eigenvalues𝜔𝜔𝑖𝑖: eigenvectors

Step A: Spectrum-Preserving Node Reduction

Global graph embedding using the first few Laplacian eigenvectors– First k eigenvectors for k-dimensional spectral graph embedding

– Key to spectral graph partitioning & data clustering [1-2]

[1] J. Lee, et al. Multiway spectral partitioning and higher-order Cheeger inequalities. JACM’14[2] R. Peng, et al. Partitioning well-clustered graphs: Spectral clustering works!." COLT’15.

𝒘𝒘𝟑𝟑

𝒘𝒘𝟐𝟐

𝒘𝒘𝟒𝟒

A naïve spectral node reduction scheme: aggregate nodes that are close to each other in the embedding spaceChallenge: too costly eigenvector computations

Spectral Aggregation: A Local Embedding Approach

Smoothers in multigrid for approximating low eigenvectors

Aggregation by spectral embedding w/ smoothed vectors:

low to high freq. components of a random vector(combination of all eigenvectors)

low freq. components after smoothing

smooth

(combination of the first few eigenvectors)

𝑋𝑋(1),𝑋𝑋(2),⋯ ,𝑋𝑋 𝐾𝐾 : K smoothed vectors obtained by running a few Gauss-Seidel (GS) relaxations w/ initial K random vectors [1]

[1] O. Livne and A. Brandt. Lean algebraic multigrid (LAMG): Fast graph Laplacian linear solver, SIAM Journal on Scientific Computing, 34(4):B499–B522, 2012.

Step B: Spectral Graph Sparsification & Scaling Find a graph proxy (sparsifier) to mimic the original graph

– w/ the same set of nodes but much fewer edges

– Key to designing fast numerical & graph algorithms

Spectral graph sparsifiers– Spectral sparsifiers preserve (Spielman & Teng. SIAM J. Comp.’11):

• Eigenvalues & eigenvectors of graph Laplacian matrices• Pair-wise distances, commute times, cuts between nodes, …• Effective resistances of resistor networks

The original graph The sparsified graph

Laplacian quadratic form measures the boundary size

Edges going out S:

1( )

0x u

=

if node u is in S

otherwise

( , )TGx L x cut S S=

S

( , , )G V E w=

0

0 0

1

1

max 1

min 11 0

max max ( , )

min min

Courant Fischer theorem:

( , )T

TGx

TGx

x

x L x cut S S

x L x cut S S

τ

τ=

==

= ⇒

−

= ⇒

S

( , , )G V E w=x1 x2

x3

x4

x5

Fiedler vector: the eigenvector for minτ

Revisiting Graph Laplacian Eigenvectors

Generalized Eigenvectors for Embedding & Sparsification [1]

max 1

Courant Fischer theorem for generalized eigenvalues

max cuts in Gmin c

maxuts in P

TG

TxP

x L xx L x

λ=

−

= ≥

Graph G Subgraph P

Max cut mismatch is bounded by 𝜆𝜆𝐦𝐦𝐦𝐦𝐦𝐦 !

[1] Z. Feng. Spectral Graph Sparsification in Nearly-Linear Time Leveraging Efficient Spectral Perturbation Analysis. DAC’16

An Optimization Framework for Edge Scaling Subgraph scaling via constrained optimizations

– Minimize 𝜆𝜆𝐦𝐦𝐦𝐦𝐦𝐦 by scaling up edge weights iteratively

– Control the decrease of 𝜆𝜆min for improving spectral similarity

– Accelerate stochastic gradient decent (SGD) iterations w/ fast Laplacian solver

( )

( )( )( )

max

max 1 2 min

(0)

, 1,..., ;

... ;

.

minimize :subject to:

n

s

G i i P i

n

n n

w

a L u L u i n

b

c λ

λ

λ

λ λ λ λ λ

λ λ

= =

= ≥ ≥ =

≥ ∆

Step C: Effective-Resistance Preserving Post Scaling Effective-resistance scaling scheme

– Globally scale up the reduced graph

– Match the original effective resistances

Effective resistance computation:

power diss. due to a unit current between p and qOur approach:

match power diss. due to a random current vector

Accuracy of Effective Resistances

Average relative errors of effective resistances

2.40% 2.40% 2.60% 2.90%3.60%

4.80%

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

5X 66X30X 146X2X 13XReduction Ratio

Relative Error

Average relative errors of effective resistance w/ different reduction ratios

fe_tooth graph (SuiteSparse Matrix Collection)

Algorithm Scalability

The worse-case algorithm complexity is 𝑶𝑶(|𝑬𝑬𝑮𝑮|𝒍𝒍𝒍𝒍𝒍𝒍(|𝑽𝑽𝑮𝑮|))

-10

0

10

20

30

40

50

60

4.4E4

1.3E53.6E5

3.3E63.8E6

4.0E6

5.3E6

9.2E6

1.2E7

2.0E7

Runt

ime(

s)

|𝐸𝐸𝐺𝐺| log |𝑉𝑉𝐺𝐺|

Conclusion

Key contribution:– A scalable approach for effective-resistance preserving spectral reduction of large graphs

Key ideas:– Node reduction based on local spectral graph embedding

– Spectral sparsification and scaling help better preserve eigenvalues & eigenvectors

Key results:– Preservation of key eigenvalues and eigenvectors in the reduced graphs

– Preservation of effective resistances between nodes on reduced graphs

Come to our poster for more details!