planted cliques, iterative thresholding and message ...yash/slides/simons.pdf · deshpande,...

Planted Cliques, Iterative Thresholding and MessagePassing Algorithms

Yash Deshpande and Andrea Montanari

Stanford University

November 5, 2013

Deshpande, Montanari Planted Cliques November 5, 2013 1 / 49

Problem Definition

Given distributions Q0;Q1,

A Set ! S � [n]

Data ! Aij �(Q1 if i ; j 2 S

Q0 otherwise.

Aij = Aji

Problem: Given A, identify S

Problem Definition

A Set ! S � [n]

Q0 otherwise.

Aij = Aji

Problem Definition

A Set ! S � [n]

Q0 otherwise.

Aij = Aji

An Example

Q1 = N(�; 1)

Q0 = N(0; 1):

An Example

A = �uSuTS + Z

∼ N(0, 1)

Data = Sparse, Low Rank + noise

An Example

A = �uSuTS + Z

∼ N(0, 1)

Data = Sparse, Low Rank + noise

Much work in statistics

Denoising:

y = x + noise

[Donoho, Jin 2004], [Arias-Castro, Candes, Durand 2011][Arias-Castro, Bubeck, Lugosi 2012]

Sparse signal recovery:

y = Ax + noise

[Candes, Romberg, Tao 2004], [Chen, Donoho 1998]

The simple example combines different structures

Denoising:

y = x + noise

y = Ax + noise

Denoising:

y = x + noise

y = Ax + noise

Our Running Example: Planted Cliques

A is “adjacency” matrix

Q1 = �+1

Q0 =12�+1 +

12��1:

S forms a clique in the graph

[Jerrum, 1992]

A is “adjacency” matrix

Q1 = �+1

Q0 =12�+1 +

12��1:

S forms a clique in the graph

[Jerrum, 1992]

I Average case version ofMAX-CLIQUE

I Communities in networks

I Average case version ofMAX-CLIQUE

I Communities in networks

How large should S be?

Let jS j = k

Size of largest clique in G�n; 1

�Second moment calculation ) k > 2 log2 n.

How large should S be?

Let jS j = k

Size of largest clique in G�n; 1

�Second moment calculation ) k > 2 log2 n.

Progress(0)

Complexity

n2 logn

2 log2 n

Exhaustive search

A Naive Algorithm

Pick k largest degree vertices of G as bS .

A Naive Algorithm

Pick k largest degree vertices of G as bS .

Analysis of NAIVE

If i =2 S :

deg(i) = Binomial�n � 1;

�) max

i =2Sdeg(i) � n

qn log n)

If i 2 S :

deg(i) = k � 1 + Binomial�n � k + 1;

�) min

i2Sdeg(i) � k � 1

2� O(

qn log n)

NAIVE works if: k � O(pn log n)

Analysis of NAIVE

If i =2 S :

�) max

i =2Sdeg(i) � n

qn log n)

If i 2 S :

�) min

2� O(

qn log n)

Analysis of NAIVE

If i =2 S :

�) max

i =2Sdeg(i) � n

qn log n)

If i 2 S :

�) min

2� O(

qn log n)

Analysis of NAIVE

If i =2 S :

�) max

i =2Sdeg(i) � n

qn log n)

If i 2 S :

�) min

2� O(

qn log n)

Progress(1)

Complexity

n2 logn

2 log2 n

√n log n

[Kučera, 1995]

Spectral Method

A = uSuTS + Z

±1±1

Hopefully v1(A) � uS

Spectral Method

A = uSuTS + Z

±1±1

Hopefully v1(A) � uS

Analysis of SPECTRAL

1pnA =

�kpn

�| {z }

eSeTS +

By standard linear algebra:

�� Zp

2� 2 =) hv1(A); eSi � 1� �(�)

SPECTRAL works if k � Cpn.

Progress(2)

Complexity

n2 logn

2 log2 n

√n log n

n2 log n

[Alon, Krivelevich and Sudakov, 1998]

Progress(2)

Complexity

n2 logn

2 log2 n

√n log n

n2 log n

[Ames, Vavasis 2009], [Feige, Krauthgamer 2000]

Progress(3)

Complexity

n2 logn

2 log2 n

√n log n

n2 log n

C√n1.65

[Dekel, Gurel-Gurevich and Peres, 2010]

Progress(3)

Complexity

n2 logn

2 log2 n

√n log n

n2 log n

C√n1.261

[Dekel, Gurel-Gurevich and Peres, 2010]

Progress(3)

Complexity

n2 logn

2 log2 n

√n log n

n2 log n

C√n1.261

[Alon, Krivelevich and Sudakov, 1998]

Phase transition in spectrum of A

If k � (1 + ")pn

Limiting Spectral Density

hv1(A); eSi � �(")

If k � (1 + ")pn

hv1(A); eSi � �(")

If k � (1� ")pn

hv1(A); eSi � n�1=2+�0(")

If k � (1 + ")pn

hv1(A); eSi � �(")

If k � (1� ")pn

hv1(A); eSi � n�1=2+�0(")

[Knowles, Yin, 2011]

Seems much harder than it looks!

I “Statistical algorithms” fail if k = n1=2��: [Feldman et al., 2012]

I r -Lovász-Schrijver fails for k � pn=2r : [Feige, Krauthgamer, 2002]

I r -Lasserre fails for k �pn

(log n)r2: [Widgerson and Meka, 2013]

Our result

Theorem (Deshpande, Montanari, 2013)

If jS j = k � (1 + ")pn=e, there exists an O(n2 log n) time algorithm that

identifies S with high probability.

I will present:1 A (wrong) heuristic analysis2 How to fix the heuristic3 Lower bounds

Our result

Iterative Thresholding

The power iteration:

v t+1 = A v t :

Improvement:

v t+1 = AFt(vt):

where Ft(v) = (ft(v1); ft(v2); � � � ; ft(vn))T

Choose ft(�) to exploit sparsity of eS

v t+1 = A v t :

Improvement:

v t+1 = AFt(vt):

v t+1 = A v t :

Improvement:

v t+1 = AFt(vt):

v t+1 = A v t :

Improvement:

v t+1 = AFt(vt):

v t+1 = A v t :

Improvement:

v t+1 = AFt(vt):

(Wrong) Analysis

v t+1i =

Aij ft(vtj ):

Aij are random �1 r.v.

Use Central Limit Theorem for v t+1i

(Wrong) Analysis

v t+1i =

Aij ft(vtj ):

Aij are random �1 r.v.

Use Central Limit Theorem for v t+1i

(Wrong) Analysis

If i =2 S :

v t+1i =

Aij ft(vtj )

� N�0;

ft(vtj )2

Letting v ti � N(0; �2t ). . .

�2t+1 =

ft(vtj )2

= Efft(�t�)2g;� � N(0; 1)

(Wrong) Analysis

If i =2 S :

v t+1i =

Aij ft(vtj )

� N�0;

ft(vtj )2

�2t+1 =

ft(vtj )2

= Efft(�t�)2g;� � N(0; 1)

(Wrong) Analysis

If i =2 S :

v t+1i =

Aij ft(vtj )

� N�0;

ft(vtj )2

�2t+1 =

ft(vtj )2

= Efft(�t�)2g;� � N(0; 1)

(Wrong) Analysis

If i =2 S :

v t+1i =

Aij ft(vtj )

� N�0;

ft(vtj )2

�2t+1 =

ft(vtj )2

= Efft(�t�)2g;� � N(0; 1)

(Wrong) AnalysisIf i 2 S :

v t+1i =

ft(vtj )

| {z }�t+1

Xj =2S

Aij ft(vtj )

| {z }�N(0;�2

�t+1 =1pn

ft(vtj )

�kpn

��1k

ft(vtj )

�= �Efft(�t + �t�)g;� � N(0; 1)

v t+1i =

ft(vtj )

| {z }�t+1

Xj =2S

Aij ft(vtj )

| {z }�N(0;�2

�t+1 =1pn

ft(vtj )

�kpn

��1k

ft(vtj )

�= �Efft(�t + �t�)g;� � N(0; 1)

v t+1i =

ft(vtj )

| {z }�t+1

Xj =2S

Aij ft(vtj )

| {z }�N(0;�2

�t+1 =1pn

ft(vtj )

�kpn

��1k

ft(vtj )

�= �Efft(�t + �t�)g;� � N(0; 1)

v t+1i =

ft(vtj )

| {z }�t+1

Xj =2S

Aij ft(vtj )

| {z }�N(0;�2

�t+1 =1pn

ft(vtj )

�kpn

��1k

ft(vtj )

�= �Efft(�t + �t�)g;� � N(0; 1)

Summarizing . . .

−4 −2 0 2 40

vti , i ∈ S

vti , i /∈ S

State Evolution

�t+1 = �E fft(�t + �t�)g�2t+1 = Efft(�t�)2g:

Using the optimal function ft(x) = e�tx��2t

�t+1 = �e�2t =2

�2t+1 = 1

State Evolution

�t+1 = �E fft(�t + �t�)g�2t+1 = Efft(�t�)2g:

Using the optimal function ft(x) = e�tx��2t

�t+1 = �e�2t =2

�2t+1 = 1

Fixed points develop below threshold!

If � > 1pe

If � < 1pe

If � > 1pe

If � < 1pe

If � > 1pe

If � < 1pe

If � > 1pe

If � < 1pe

If � > 1pe

If � < 1pe

Analysis is wrong but. . .

. . . so we modify the algorithm.

What algorithm?

Slight modification to iterative scheme:

(v ti )i2[n] ! (v ti!j)i ;j2[n]

v t+1i!j =

X6̀=i ;j

Ai` ft(vt`!i ):

Analysis is exact as n!1.

What algorithm?

(v ti )i2[n] ! (v ti!j)i ;j2[n]

v t+1i!j =

X6̀=i ;j

Ai` ft(vt`!i ):

What algorithm?

(v ti )i2[n] ! (v ti!j)i ;j2[n]

v t+1i!j =

X6̀=i ;j

Ai` ft(vt`!i ):

What algorithm?

(v ti )i2[n] ! (v ti!j)i ;j2[n]

v t+1i!j =

X6̀=i ;j

Ai` ft(vt`!i ):

Fixing the heuristic

LemmaLet (ft(z))t�0 be a sequence of polynomials. Then, for every fixed t, andbounded, continuous function : R! R the following limit holds inprobability:

limn!1

(v ti!j) = �Ef (�t + �t�)g;

limn!1

Xi2[n]nS

(v ti!j) = Ef (�t�)g;

where � � N(0; 1).

Proof Technique

Key ideas:

Expand v ti!j for polynomial ft(�)

Wrong analysis works if A! At

Proof Technique

Key ideas:

Proof Technique

Key ideas:

Proof Technique - Expanding v t

Let ft(x) = x2; v0i!j = 1

v1i!j =

Xk 6=j

Aik :i

v2i!j =

Xk 6=j

Aik(v1k!i )

=Xk 6=j

�X` 6=i

��Xm 6=i

�=Xk 6=j

X` 6=i

Xm 6=i

AikAk`Akm:

v2i!j =

Xk 6=j

Aik(v1k!i )

=Xk 6=j

�X` 6=i

��Xm 6=i

�=Xk 6=j

X` 6=i

Xm 6=i

AikAk`Akm:

v2i!j =

Xk 6=j

Aik(v1k!i )

=Xk 6=j

�X` 6=i

��Xm 6=i

�=Xk 6=j

X` 6=i

Xm 6=i

AikAk`Akm:

Proof Technique

v t+1i!j =

Xk 6=i

Aik ft(vtk!i ):

�t+1i!j =

Xk 6=i

Atik ft(�

tk!i ):

Proof Technique - a Combinatorial Lemma

v ti!j =X

T2T ti!j

A(T )Γ(T )v0(T )

where T ti!j consists rooted, labeled trees that:

1 have maximum depth t.2 do not backtrack.

(Similarly for the �ti!j)

Proof Technique - Moment Method

v t+1i!j =

Xk 6=i

Aik ft(vtk!i ):

�t+1i!j =

Xk 6=i

Atik ft(�

tk!i ):

limn!1 Moments of v t+1 = Moments of �t+1.

Proof Technique - Moment Method

v t+1i!j =

Xk 6=i

Aik ft(vtk!i ):

�t+1i!j =

Xk 6=i

Atik ft(�

tk!i ):

limn!1 Moments of v t+1 = Moments of �t+1.

Progress(4)

Complexity

n2 logn

2 log2 n

2√n log n

n2 log n

C√n1.261

Spectral threshold =√n

Is this threshold fundamental?

Rest of the talk: perhaps

The “Hidden Set” Problem

Given Gn = ([n];En)

A Set ! S � [n]

Data ! Aij �(Q1 if i ; j 2 S ;

Q0 otherwise.

Aij ∼ Q1

Aij ∼ Q0

Problem: Given edge labels (Aij)(i ;j)2En , identify S

Given Gn = ([n];En)

A Set ! S � [n]

Q0 otherwise.

Aij ∼ Q1

Aij ∼ Q0

Given Gn = ([n];En)

A Set ! S � [n]

Q0 otherwise.

“Local” Algorithms

A t-local algorithm computes:

Estimate at i :

bu(i) = F(ABall(i ;t))

The Sparse Graph Analogue

Gn = ([n];En); n � 1 satisfies:I locally tree-likeI regular degree ∆

FurtherI Q1 = �+1, Q0 = 1

2�+1 + 12��1

Aij ∼ Q1

Aij ∼ Q0

What can local algorithms achieve?

2�+1 + 12��1

Aij ∼ Q1

Aij ∼ Q0

2�+1 + 12��1

Aij ∼ Q1

Aij ∼ Q0

2�+1 + 12��1

Aij ∼ Q1

Aij ∼ Q0

What can we hope for?

If jS j = C np∆:

bSnaive = Random set of size jS j

) 1nEfbSnaive4Sg = Θ

�1p∆

What can we hope for?

If jS j = C np∆:

Poisson bound ) for any local algorithm:

1nEfbS4Sg � e�C

A result for local algorithms. . .

Theorem (Deshpande, Montanari, 2013)Let Gn converge locally to ∆�regular tree:If jS j � (1 + ") np

e∆there exists a local algorithm achieving

1nEfS4bSg � e�Θ(

p∆):

Conversely, if jS j � (1� ") npe∆

every local algorithm suffers

1nEfS4bSg � Θ

�1p∆

With ∆ = n � 1 we recover the complete graph result!

p∆):

1nEfS4bSg � Θ

�1p∆

p∆):

1nEfS4bSg � Θ

�1p∆

To conclude. . .

I Message-passing algorithm performs “weighted” counts ofnon-reversing trees

I Such structures have been used elsewhere:� Clustering sparse networks: [Krzakala et al. 2013]� Compressed sensing: [Bayati, Lelarge, Montanari 2013]

To conclude. . .

I What is the “dense” analogue for local algorithms?

I What about other structural properties?

Thank you!

To conclude. . .

Thank you!

To conclude. . .

Thank you!

planted cliques, iterative thresholding and message ...yash/slides/simons.pdf · deshpande,...

Documents