NEIGHBORHOOD FORMATION AND ANOMALY DETECTION IN BIPARTITE GRAPHS

Jimeng Sun, Huiming Qu,

Deepayan Chakrabarti & Christos Faloutsos

Presented ByBhavana Dalvi

OUTLINE

Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

Author Paper graph

Authors Papers

a

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

Author Paper graph

Authors Papers

aWhich authors are most related to ‘a’ ?

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

Author Paper graph

Authors Papers

aWhich authors are most related to ‘a’ ?

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

Author Paper graph

Authors Papers

aWhich authors are most related to ‘a’ ?

0.8b

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

Author Paper graph

Authors Papers

aWhich authors are most related to ‘a’ ?

0.8

0.6

0.2

0.4

b

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

Author Paper graph

Authors Papers

a

Which is the uncommon paper written by ‘a’ ?

0.8

0.6

0.2

0.4

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

Author Paper graph

Authors Papers

a

Which is the uncommon paper written by ‘a’ ?

0.8

0.6

0.2

0.4

BIPARTITE GRAPHS AND INTERESTING QUESTIONS

P2P Network

10

users

files

Which users have similar preferences as a particular user?

Jimeng Sun’s presentation at ICDM 2005

Which files are downloaded by users with very different preferences?

OUTLINE

Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work

Neighborhood formation (NF)Input : query node q in V1

Output : relevance scores of all the nodes in V1 to q

Anomaly detection (AD)Input : query node q in V1, Output : normality scores for nodes in V2 that link to q

PROBLEM DEFINITION

V1 V2

q

E

OUTLINE

Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work

NEIGHBORHOOD FORMATION

Relevance (b, q) (# short length paths from q to b)

b

q

The connection that links only b and q brings more relevance than the connection which links b, q and other nodes.

b

q

EXACT NF ALGORITHM : RANDOM WALK WITH RESTART

Input : a graph G and a query node q

Output : relevance scores to q Construct the transition matrix where

every node in the graph becomes a state every state has a restart probability c to jump back to the query node q. transition probability

Find the steady-state probability u which is the relevance score of all the nodes to q

q

cc c

c

c

Jimeng Sun’s presentation at ICDM 2005

FINDING STEADY STATE PROBABILITIES

|V1| = k , |V2| = n M : k*n matrix representing weighted graph G Adjacency matrix : PA = col_norm(MA) qA : transform query node ‘a’ to (k+n)*1 vector

where only ath column has 1 and rest are 0. uA : steady state probability vector with restart

probability c

Bipartite structure :

k << n then savings are significant

EXTENSIONS TO NF ALGORITHM

Parallel NF If multiple queries, computation can be done in

parallel.

Approximate NF Cluster the nodes in to k partitions

(preprocessing) Given query node q, find partition Gi it belongs to Run Exact NF algorithm only on Gi

Set relevance = 0 for nodes not in Gi

OUTLINE

Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work

ANOMALY DETECTION

A node x in V2 is normal if

Nodes in V1 that links to x

are in same neighbourhood. e.g. V1

V2

V1 V2

low normalityhigh normality

xx

ANOMALY DETECTION ALGORITHM

Input : node t in V2, Bipartite transition matrix P,

Output : Normality score(t)

1. Set St = neighbours of t in V1

2. RSt : Pairwise relevance scores for nodes in St

3. Normality score ns(t) = function (RSt) e.g. mean over non-diagonal elements in

RSt

OUTLINE

Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work

DATASETS

datasets |V1| |V2| |E| Avgdeg (V1)

Avgdeg (V2)

Conference-Author (CA)

2687 288K 662K 510 5

Author-Paper (AP)

316K 472K 1M 3 2

IMDB 553K 204K 2.2M 4 11

DO THE NEIGHBORHOODS MAKE SENSE?

rele

van

ce s

core

rele

vanc

e sc

ore

rele

vanc

e sc

ore

most relevant neighbors most relevant neighbors

The nodes (x-axis) with the highest relevance scores (y-axis) are indeed very relevant to the query node.

HOW ACCURATE IS THE APPROXIMATE NF?

neighborhood size = 20 num of partitions = 10

Precision = fraction of overlaps between ApprNF and NF among top k neighbors

The precision drops slowly while increasing the number of partition The precision remain high for a wide range of neighborhood size

DO THE ANOMALIES MAKE SENSE?

avg

. nor

mal

ity

scor

e

Injection : • Inject 100 nodes in V2 connecting k nodes each in V1

where k = avg. degree of nodes in V2

• Nodes in V1 are randomly picked such that degree = 10 * avg. degree of nodes in V1

• Assumption : will induce connections across neighbourhoods

WHAT ABOUT THE COMPUTATIONAL COST?

Computational cost drops significantly even with small increment in number of partitions

OUTLINE

Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work

RELATEDWORK

Random walk on Graphs Page-Rank [ISDN 1998], Topic Sensitive Page-Rank [WWW 2002]

Outlier detection Outlier detection in high dimensional data : Aggarwal

and Yu [SIGMOD 2001] Outlier Detection Using Random Walks [ICTAI 2006]

Find outlier clusters

Graph partitioning : METIS package Spectral clustering methods Neighbourhoods can become personalized clusters

OUTLINE

Motivation Problem Definition Neighborhood formation Anomaly detection Experiments Related work Conclusion and future work

CONCLUSIONS AND FUTURE WORK

Solution to two problems for Bipartite Graphs Neighborhood Formation (NF) Anomaly Detection (AD)

Random walk with restart along with graph partitioning can be used to solve NF efficiently.

AD can be done based on relevance scores generated by NF

Experiments on real datasets show good results.

Proximity Tracking on Time-Evolving Graphs (SIAM 2008 paper) Defines proximity scores in dynamic setting. Efficient incremental updates

THANK YOU