streaming graph partitioning for large distributed...
TRANSCRIPT
![Page 1: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/1.jpg)
Streaming graph partitioning for large distributed graphs
Daniel Spanier
June 5, 2015
Daniel Spanier Streaming graph partitioning June 5, 2015 1 / 40
![Page 2: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/2.jpg)
Overview
1 IntroductionMotivationBalanced graph partitioning
2 HeuristicsCompetitorsStream orderingDatasets
3 FENNEL
4 Conclusion
Daniel Spanier Streaming graph partitioning June 5, 2015 2 / 40
![Page 3: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/3.jpg)
Motivation
Lets take a look at Pregel:
Pregel has to load the vertices
and distribute them over its machines
then graph processing can start
Daniel Spanier Streaming graph partitioning June 5, 2015 3 / 40
![Page 4: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/4.jpg)
Motivation
Pregel uses a hash function h : V ! {1, . . . k}
Now each machine has the same amount of vertices.Daniel Spanier Streaming graph partitioning June 5, 2015 4 / 40
![Page 5: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/5.jpg)
Motivation
doesn’t minimize number of edges cut
higher communication cost between machines afterwards
Daniel Spanier Streaming graph partitioning June 5, 2015 5 / 40
![Page 6: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/6.jpg)
Motivation
doesn’t minimize number of edges cut
higher communication cost between machines afterwards
But: Pregel supports customized graph partitioning ! use balanced graphpartitioning
Daniel Spanier Streaming graph partitioning June 5, 2015 6 / 40
![Page 7: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/7.jpg)
Balanced k-graph partitioning
Let k be the number of partitions we want to have.The goal is to minimize the cross partition edges and keeping the numberof nodes in each partition even.
But: This is NP-hard, so in practise, only approximations are feasible.
Daniel Spanier Streaming graph partitioning June 5, 2015 7 / 40
![Page 8: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/8.jpg)
Graph partitioning variants
We have two di↵erent approaches:
Daniel Spanier Streaming graph partitioning June 5, 2015 8 / 40
![Page 9: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/9.jpg)
Graph partitioning
- o✏ine: know all vertices beforehand, then decide their places
- streaming: load the graph vertex after vertex and decide vertex placeimmediately
O✏ine algorithms are better at minimizing the amount of cut edges,streaming algorithm can be used during the same time pregel loads thegraph into main memory.
Daniel Spanier Streaming graph partitioning June 5, 2015 9 / 40
![Page 10: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/10.jpg)
O✏ine graph partitioning
METIS is widely used
combines various graph partition algorithms
needs access to all vertices at once, so it cannot be used during theloading phase of Pregel
We will compare the results of the streaming graph partitioningalgorithms with those of METIS
Daniel Spanier Streaming graph partitioning June 5, 2015 10 / 40
![Page 11: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/11.jpg)
Online graph partitioning
Let k be the number of partitions we want, P = (P1
, . . .Pk
) the partitionsand G = (V ,E ) the graph with |V | = n, |E | = m
Heuristics decide, in which partition Pi
a vertex v 2 V is put.We will see two kinds of heuristics: with a bu↵er and without one
Daniel Spanier Streaming graph partitioning June 5, 2015 11 / 40
![Page 12: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/12.jpg)
Heuristics
Hashing
We use a fairly simple hash-function H: V ! {1, . . . k} :H(v) = (v mod k) + 1
Hashing is mainly used in practise, because every machine knows where anode has been distributed to without the need of a mapping table.
Daniel Spanier Streaming graph partitioning June 5, 2015 12 / 40
![Page 13: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/13.jpg)
Heuristics
Chunking
Let C be the capacity of a Partition.Then: divide stream into chunks of size C and fill partitions in orderindex(v) = (v div C ) + 1
Daniel Spanier Streaming graph partitioning June 5, 2015 13 / 40
![Page 14: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/14.jpg)
Heuristics
Deterministic Greedy
Assign v to the partition where it has the most edges.
Let t be the time v arrives, N(v) be the set of vertices v neighbours andw a weight function penalizing partitions with too many vertices.index(v) = argmax
i2[k]{|P
i
\ N(v)| ⇤ w(i , t)}
Daniel Spanier Streaming graph partitioning June 5, 2015 14 / 40
![Page 15: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/15.jpg)
Heuristics
Deterministic Greedy
Assign v to the partition where it has the most edges.Let t be the time v arrives, N(v) be the set of vertices v neighbours andw a weight function penalizing partitions with too many vertices.index(v) = argmax
i2[k]{|P
i
\ N(v)| ⇤ w(i , t)}
w(i , t) = 1 for Unweighted Deterministic Greedy
w(i , t) = 1� |Pi
|C
for Linear Deterministic Greedy
w(i , t) = 1� exp{|Pi
|� C} for Exponentially Deterministic Greedy
Daniel Spanier Streaming graph partitioning June 5, 2015 15 / 40
![Page 16: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/16.jpg)
Heuristics
This heuristic uses a Bu↵er of size C and assumes a way to di↵erentiatebetweeen high-degree and low-degree nodes
Avoid Big
We maintain the bu↵er and a threshold on large nodes.If the bu↵er is filled or the threshold is reached we greedily assign alllow-degree nodes in the bu↵er.If only high-degree nodes remain, assign them with deterministic greedy.
Daniel Spanier Streaming graph partitioning June 5, 2015 16 / 40
![Page 17: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/17.jpg)
Stream ordering
Stream ordering means the order in which the vertices arrive at thestreaming algorithm.Depending on the ordering, di↵erent heuristics achieve better or worseresults ! we have to specify the ordering
Random: Standard Ordering in literature
Breadth-First-Search ordering
Depth-First-Search ordering
Daniel Spanier Streaming graph partitioning June 5, 2015 17 / 40
![Page 18: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/18.jpg)
Datasets
Marvel Comics social network, which resembles real social networks|V | = 6, 486, |E | = 427, 018
PL, synthetic graphs generated by a power-law graph generator withclusteringexample P1000:|V | = 1000, |E | = 9, 878
Daniel Spanier Streaming graph partitioning June 5, 2015 18 / 40
![Page 19: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/19.jpg)
Evaluation
Marvel, k = 8
Daniel Spanier Streaming graph partitioning June 5, 2015 19 / 40
![Page 20: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/20.jpg)
Evaluation
PL1000, k = 4
Daniel Spanier Streaming graph partitioning June 5, 2015 20 / 40
![Page 21: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/21.jpg)
Evaluation
Name Random BFS DFSAvoid Big -46.4 -27.3 -38.6Chunking 0.7 37.6 35.7Deterministic Greedy 45.4 57.7 54.7Exp. Det. Greedy 47.5 59.4 56.2Hashing -1.7 -1.9 -2.1Linear Det. Greedy 75.3 76 73
Daniel Spanier Streaming graph partitioning June 5, 2015 21 / 40
![Page 22: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/22.jpg)
Results
Linear Deterministic Greedy had the highest average gain, no matterwhat streaming ordering was used
Nearly every heuristic is an improvement
Except Avoid Big with negative improvement; this heuristic shouldn’tbe used
Daniel Spanier Streaming graph partitioning June 5, 2015 22 / 40
![Page 23: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/23.jpg)
Results
Why is Linear Deterministic Greedy better than the others?
Deterministic Greed
index(v) = argmaxi2[k]
{|Pi
\ N(v)| ⇤ w(i , t)}
w(i , t) = 1 for Unweighted Det. Greedy
w(i , t) = 1� |Pi
|C
for Linear Det. Greedy
w(i , t) = 1� exp{|Pi
|� C} for Exponentially Det. Greedy
Daniel Spanier Streaming graph partitioning June 5, 2015 23 / 40
![Page 24: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/24.jpg)
Results
Why is Linear Deterministic Greedy better than the others?
Unweighted Det. Greedy only indicates a already full partition
Exponential Det. Greedy indicates a full partition only nearly at thelimit
Linear Det. Greedy prefers less loaded partitions
Daniel Spanier Streaming graph partitioning June 5, 2015 24 / 40
![Page 25: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/25.jpg)
FENNEL
FENNEL combines two widely used families of heuristics:
place newly arrived vertices in Partition with the largest number ofneighbours
place newly arrived vertices in Partition with the least non neighbours
Daniel Spanier Streaming graph partitioning June 5, 2015 25 / 40
![Page 26: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/26.jpg)
FENNEL
Let P = (P1
, ...Pk
) be all k > 0 partitions, Pi
Partition i .Then we define a global objective function:
f (P) = cout
(P) + cin
(P)
Daniel Spanier Streaming graph partitioning June 5, 2015 26 / 40
![Page 27: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/27.jpg)
FENNEL
cout
(P) = #edges cut
is the inter-partition cost.But what to do with c
in
(P)?
Daniel Spanier Streaming graph partitioning June 5, 2015 27 / 40
![Page 28: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/28.jpg)
FENNEL
But what to do with cin
(P)?
Focus on individual partitions: cin
(P) =P
k
i=1
cin
(Pi
)
Size of Pi
should be approximately n
k
Daniel Spanier Streaming graph partitioning June 5, 2015 28 / 40
![Page 29: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/29.jpg)
FENNEL
We take a look at function family c(x) = ↵ ⇤ x�
� lets us control the importance of imbalance in the partition size
� = 1 means ignoring imbalance of partition sizes
� = 2 would diminish the importance of cout
(P)
Most of the time, the authors stuck to � = 1.5
Daniel Spanier Streaming graph partitioning June 5, 2015 29 / 40
![Page 30: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/30.jpg)
FENNEL
We take a look at function family c(x) = ↵ ⇤ x�
↵ = m ⇤ k
��1
n
� , with |E | = mwith that we get:
f (P) = cout
(P) +P
k
i=1
cin
(Pi
)
= #edges�cut
m
+ 1
k
Pk
i=1
( |Pi
|n/k )
�
Daniel Spanier Streaming graph partitioning June 5, 2015 30 / 40
![Page 31: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/31.jpg)
FENNEL
The graph partition problem with this objective function becomes:
minP
(#edges�cut
m
+ 1
k
Pk
i=1
( |Pi
|n/k )
�)
But this isn’t doable in a streaming algorithm!
Daniel Spanier Streaming graph partitioning June 5, 2015 31 / 40
![Page 32: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/32.jpg)
FENNEL
Solution: We greedily decide, to which Pi
we assign an incoming vertex v
f ((P1
, . . .Pi
[ {v}, . . .Pk
)) f ((P1
, . . .Pj
[ {v}, . . .Pk
) , 8j 2 {1, . . . k}
To stop overloading, we can add a threshold t:If |P
i
| > t ⇤ n
k
, Pi
can’t be chosen for v
Daniel Spanier Streaming graph partitioning June 5, 2015 32 / 40
![Page 33: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/33.jpg)
FENNEL
Transform minimization problem into maximization:g((P
1
, . . .Pi
[ {v}, . . .Pk
)) � g((P1
, . . .Pj
[ {v}, . . .Pk
) , 8j 2 {1, . . . k}
This leads to:G (v ,P
i
) = |N(v) \ Pi
|� ↵|Pi
|��1
Daniel Spanier Streaming graph partitioning June 5, 2015 33 / 40
![Page 34: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/34.jpg)
FENNEL
G (v ,Pi
) = |N(v) \ Pi
|� ↵�|Pi
|��1
For � = 1 this is Deterministic Unweighted Greedy.For � = 2 = 1
↵ FENNEL would place vertices to the partitions with theleast number of non-neighbours
Daniel Spanier Streaming graph partitioning June 5, 2015 34 / 40
![Page 35: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/35.jpg)
FENNEL
Now we compare FENNEL with Linear Det. Greedy and METIS (k = 32):
Name BFS % edges cut BFS max partition loadLinear Det. Greedy 34 % 1.01FENNEL 14% 1.10METIS 8% 1.00
Daniel Spanier Streaming graph partitioning June 5, 2015 35 / 40
![Page 36: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/36.jpg)
FENNEL
Now we compare FENNEL with Linear Det. Greedy and METIS (k = 32):
Name RANDOM % edges cut RANDOM part. loadLinear Det. Greedy 40 % 1.00FENNEL 14% 1.02METIS 8% 1.02
Daniel Spanier Streaming graph partitioning June 5, 2015 36 / 40
![Page 37: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/37.jpg)
FENNEL
. . . and with METIS alone:
Daniel Spanier Streaming graph partitioning June 5, 2015 37 / 40
![Page 38: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/38.jpg)
FENNEL
Remember Pegel and others use a hash heuristic instead of FENNELduring the graph loading;now if we use FENNEL instead before a PageRank computation onLiveJournal:
Daniel Spanier Streaming graph partitioning June 5, 2015 38 / 40
![Page 39: Streaming graph partitioning for large distributed graphsresources.mpi-inf.mpg.de/d5/teaching/ss15/msga/ss15-msga-partitioning.pdfMotivation Lets take a look at Pregel: Pregel has](https://reader033.vdocuments.us/reader033/viewer/2022041915/5e6975c5bb7b2f2a5b02388c/html5/thumbnails/39.jpg)
Conclusion
Despite a single pass, FENNEL can achieve results comparable toMETIS, but is much faster.
We have seen that using FENNEL actually improves graph operationslike PageRank.
This improvement stems entirely from the reduced networkcommunication.
Daniel Spanier Streaming graph partitioning June 5, 2015 39 / 40