![Page 1: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/1.jpg)
CS267, Spring 2014April 15, 2014
Parallel Graph Algorithms
Aydın Buluç[email protected]
http://gauss.cs.ucsb.edu/~aydin/Lawrence Berkeley National Laboratory
Slide acknowledgments: K. Madduri, J. Gilbert, D. Delling, S. Beamer
![Page 2: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/2.jpg)
Graph Preliminaries
n=|V| (number of vertices)m=|E| (number of edges)D=diameter (max #hops between any pair of vertices)• Edges can be directed or undirected, weighted or not.• They can even have attributes (i.e. semantic graphs)• Sequences of edges <u1,u2>, <u2,u3>, … ,<un-1,un> is a
path from u1 to un. Its length is the sum of its weights.
Define: Graph G = (V,E)- a set of vertices and a set of
edges between vertices
EdgeVertex
![Page 3: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/3.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 4: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/4.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 5: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/5.jpg)
Road networks, Point-to-point shortest paths: 15 seconds (naïve) 10 microseconds
Routing in transportation networks
H. Bast et al., “Fast Routing in Road Networks with Transit Nodes”, Science 27, 2007.
![Page 6: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/6.jpg)
• The world-wide web can be represented as a directed graph– Web search and crawl: traversal– Link analysis, ranking: Page rank and HITS– Document classification and clustering
• Internet topologies (router networks) are naturally modeled as graphs
Internet and the WWW
![Page 7: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/7.jpg)
• Reorderings for sparse solvers– Fill reducing orderings
Partitioning, eigenvectors– Heavy diagonal to reduce pivoting (matching)
• Data structures for efficient exploitation of sparsity• Derivative computations for optimization
– graph colorings, spanning trees• Preconditioning
– Incomplete Factorizations– Partitioning for domain decomposition– Graph techniques in algebraic multigrid
Independent sets, matchings, etc.– Support Theory
Spanning trees & graph embedding techniques
Scientific Computing
B. Hendrickson, “Graphs and HPC: Lessons for Future Architectures”, http://www.er.doe.gov/ascr/ascac/Meetings/Oct08/Hendrickson%20ASCAC.pdf
Image source: Yifan Hu, “A gallery of large graphs”
10
13
2
4
5
6
7
8
9
G+(A)[chordal]
![Page 8: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/8.jpg)
• Graph abstractions are very useful to analyze complex data sets.
• Sources of data: petascale simulations, experimental devices, the Internet, sensor networks
• Challenges: data size, heterogeneity, uncertainty, data quality
Large-scale data analysis
Astrophysics: massive datasets, temporal variations
Bioinformatics: data quality, heterogeneity
Social Informatics: new analytics challenges, data uncertainty
Image sources: (1) http://physics.nmt.edu/images/astro/hst_starfield.jpg (2,3) www.visualComplexity.com
![Page 9: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/9.jpg)
Image Source: Nexus (Facebook application)
Graph-theoretic problems in social networks
– Targeted advertising: clustering and centrality
– Studying the spread of information
![Page 10: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/10.jpg)
Network Analysis for Neurosciences
Graph-theoretical models are used to predict the course of degenerative illnesses like Alzheimer’s.
Some disease indicators:- Deviation from small-world property- Emergence of “epicenters” with disease-associated patterns
Vertices: ROI (regions of interest) Edges: structural/functional connectivity
![Page 11: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/11.jpg)
Research in Parallel Graph AlgorithmsApplication
AreasMethods/Problems
ArchitecturesGraph Algorithms
Traversal
Shortest Paths
Connectivity
Max Flow
…
…
…
GPUs, FPGAs
x86 multicoreservers
Massively multithreadedarchitectures
(Cray XMT, Sun Niagara)
Multicore clusters
(NERSC Hopper)
Clouds(Amazon EC2)
Social NetworkAnalysis
WWW
Computational Biology
Scientific Computing
Engineering
Find central entitiesCommunity detection
Network dynamics
Data size
ProblemComplexity
Graph partitioningMatchingColoring
Gene regulationMetabolic pathways
Genomics
MarketingSocial Search
VLSI CADRoute planning
![Page 12: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/12.jpg)
Characterizing Graph-theoretic computations
• graph sparsity (m/n ratio)• static/dynamic nature• weighted/unweighted, weight distribution• vertex degree distribution• directed/undirected• simple/multi/hyper graph• problem size• granularity of computation at nodes/edges• domain-specific characteristics
• paths• clusters• partitions• matchings• patterns• orderings
Input data
Problem: Find ***
Factors that influence choice of algorithm
Graph kernel
• traversal• shortest path algorithms• flow algorithms• spanning tree algorithms• topological sort …..
Graph problems are often recast as sparse linear algebra (e.g., partitioning) or linear programming (e.g., matching) computations
![Page 13: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/13.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 14: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/14.jpg)
• Many PRAM graph algorithms in 1980s.• Idealized parallel shared memory system model• Unbounded number of synchronous processors;
no synchronization, communication cost; no parallel overhead
• EREW (Exclusive Read Exclusive Write), CREW (Concurrent Read Exclusive Write)
• Measuring performance: space and time complexity; total number of operations (work)
The PRAM model
![Page 15: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/15.jpg)
• Pros– Simple and clean semantics.– The majority of theoretical parallel algorithms are designed
using the PRAM model.– Independent of the communication network topology.
• Cons– Not realistic, too powerful communication model.– Communication costs are ignored.– Synchronized processors.– No local memory.– Big-O notation is often misleading.
PRAM Pros and Cons
![Page 16: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/16.jpg)
• Prefix sums • Symmetry breaking• Pointer jumping• List ranking• Euler tours• Vertex collapse• Tree contraction[some covered in the “Tricks with Trees” lecture]
Building blocks of classical PRAM graph algorithms
![Page 17: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/17.jpg)
Static case• Dense graphs (m = Θ(n2)): adjacency matrix commonly used.• Sparse graphs: adjacency lists, compressed sparse matrices
Dynamic• representation depends on common-case query• Edge insertions or deletions? Vertex insertions or deletions?
Edge weight updates?• Graph update rate• Queries: connectivity, paths, flow, etc.
• Optimizing for locality a key design consideration.
Data structures: graph representation
![Page 18: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/18.jpg)
Compressed sparse rows (CSR) = cache-efficient adjacency lists
Graph representations
1
3
24
1 3 2 2 4Adjacencies
(row pointers in CSR)1 3 3 4 6Index intoadjacency array
12
26
7
14
19
1
2
3
4
1 12 3 26
2 19
2 14 4 7
12 26 19 14 7Weights
(column ids in CSR)
(numerical values in CSR)
![Page 19: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/19.jpg)
• Each processor stores the entire graph (“full replication”)• Each processor stores n/p vertices and all adjacencies
out of these vertices (“1D partitioning”)• How to create these “p” vertex partitions?
– Graph partitioning algorithms: recursively optimize for conductance (edge cut/size of smaller partition)
– Randomly shuffling the vertex identifiers ensures that edge count/processor are roughly the same
Distributed graph representations
![Page 20: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/20.jpg)
• Consider a logical 2D processor grid (pr * pc = p) and the matrix representation of the graph
• Assign each processor a sub-matrix (i.e, the edges within the sub-matrix)
2D checkerboard distribution
0 7
5
3
8
2
4 6
1x x x
x
x x
x x x
x x x
x x
x x x
x x x
x x x x
9 vertices, 9 processors, 3x3 processor grid
Flatten Sparse matrices
Per-processor local graph representation
![Page 21: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/21.jpg)
• Implementations are typically array-based for performance (e.g. CSR representation).
• Concurrency = minimize synchronization (span)• Where is the data? Find the distribution that
minimizes inter-processor communication.• Memory access patterns
– Regularize them (spatial locality)– Minimize DRAM traffic (temporal locality)
• Work-efficiency– Is (Parallel time) * (# of processors) = (Serial work)?
High-performance graph algorithms
![Page 22: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/22.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 23: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/23.jpg)
Graph traversal: Depth-first search (DFS)
0 7
5
3
8
2
4 6
1
9sourcevertex
9
8
7
2 3 4
6
5
Parallelizing DFS is a bad idea: span(DFS) = O(n)
J.H. Reif, Depth-first search is inherently sequential. Inform. Process. Lett. 20 (1985) 229-234.
1
preordervertex number
![Page 24: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/24.jpg)
Graph traversal : Breadth-first search (BFS)
0 7
5
3
8
2
4 6
1
9sourcevertex
Input:Output:1
1
1
2
2 3 3
4
4
distance from source vertex
Memory requirements (# of machine words):• Sparse graph representation: m+n• Stack of visited vertices: n• Distance array: n
Breadth-first search is a very important building block for other parallel graph algorithms such as (bipartite) matching, maximum flow, (strongly) connected components, betweenness centrality, etc.
![Page 25: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/25.jpg)
1. Expand current frontier (level-synchronous approach, suited for low diameter graphs)
Parallel BFS Strategies
0 7
5
3
8
2
4 6
1
9source vertex
2. Stitch multiple concurrent traversals (Ullman-Yannakakis approach, suited for high-diameter graphs)
• O(D) parallel steps• Adjacencies of all
vertices in current frontier are visited in parallel
0 7
5
3
8
2
4 6
1
9source vertex
• path-limited searches from “super vertices”
• APSP between “super vertices”
![Page 26: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/26.jpg)
Performance observations of the level-synchronous algorithm
Phase #
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Per
cent
age
of to
tal
0
10
20
30
40
50
60 # of vertices in frontier arrayExecution time
Youtube social network
When the frontier is at its peak, almost all edge examinations “fail” to claim a child
Synthetic network
![Page 27: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/27.jpg)
Classical (top-down) algorithm is optimal in worst case, but pessimistic for low-diameter graphs (previous slide).
Bottom-up BFS algorithm
Direction Optimization: - Switch from top-down to
bottom-up search - When the majority of the
vertices are discovered.[Read paper for exact heuristic]
Scott Beamer, Krste Asanović, and David Patterson, "Direction-Optimizing Breadth-First Search", Int. Conf. on High Performance Computing, Networking, Storage and Analysis (SC), 2012
![Page 28: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/28.jpg)
1
2
3
4 7
6
5
AT
1
7
71from
to
Breadth-first search in the language of linear
algebra
![Page 29: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/29.jpg)
1
2
3
4 7
6
5
XAT
1
7
71from
to
ATX
1
1
1
1
1parents:
Replace scalar operationsMultiply -> selectAdd -> minimum
![Page 30: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/30.jpg)
1
2
3
4 7
6
5
X
4
2
2
AT
1
7
71from
to
ATX
2
4
4
2
24
Select vertex withminimum label as parent
1
1parents:4
2
2
![Page 31: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/31.jpg)
1
2
3
4 7
6
5
X
3
AT
1
7
71from
to
ATX
3
5
7
3
1
1parents:4
2
2
5
3
![Page 32: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/32.jpg)
XAT
1
7
71from
to
ATX
6
1
2
3
4 7
6
5
![Page 33: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/33.jpg)
ALGORITHM:1. Find owners of the current frontier’s adjacency [computation]2. Exchange adjacencies via all-to-all. [communication]3. Update distances/parents for unvisited vertices. [computation]
x
1 2
3
4 7
6
5
AT frontier
x
1D Parallel BFS algorithm
![Page 34: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/34.jpg)
x
1 2
3
4 7
6
5
AT frontier
x
ALGORITHM:1. Gather vertices in processor column [communication]2. Find owners of the current frontier’s adjacency [computation]3. Exchange adjacencies in processor row [communication]4. Update distances/parents for unvisited vertices. [computation]
2D Parallel BFS algorithm
![Page 35: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/35.jpg)
5040 10008 20000 40000
2
4
6
8
10
12
1D Flat MPI 2D Flat MPI1D Hybrid 2D Hybrid
Number of coresC
om
m.
tim
e (
seco
nds)
5040 10008 20000 40000
4
8
12
16
20
1D Flat MPI 2D Flat MPI1D Hybrid 2D Hybrid
Number of cores
GTEPS
• NERSC Hopper (Cray XE6, Gemini interconnect AMD Magny-Cours)• Hybrid: In-node 6-way OpenMP multithreading• Kronecker (Graph500): 4 billion vertices and 64 billion edges.
BFS Strong Scaling
A. Buluç, K. Madduri. Parallel breadth-first search on distributed memory systems. Proc. Supercomputing, 2011.
![Page 36: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/36.jpg)
• ORNL Titan (Cray XK6, Gemini interconnect AMD Interlagos)• Kronecker (Graph500): 16 billion vertices and 256 billion edges.
Direction optimizing BFS with 2D decomposition
Scott Beamer, Aydın Buluç, Krste Asanović, and David Patterson, "Distributed Memory Breadth-First Search Revisited: Enabling Bottom-Up Search”, MTAAP’13, IPDPSW, 2013
![Page 37: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/37.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 38: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/38.jpg)
Parallel Single-source Shortest Paths (SSSP) algorithms• Famous serial algorithms:
– Bellman-Ford : label correcting - works on any graph– Dijkstra : label setting – requires nonnegative edge weights
• No known PRAM algorithm that runs in sub-linear time and O(m+n log n) work
• Ullman-Yannakakis randomized approach • Meyer and Sanders, ∆ - stepping algorithm
• Chakaravarthy et al., clever combination of ∆ - stepping and direction optimization (BFS) on supercomputer-scale graphs.
V. T. Chakaravarthy, F. Checconi, F. Petrini, Y. Sabharwal “Scalable Single Source Shortest Path Algorithms for Massively Parallel Systems ”, IPDPS’14
U. Meyer and P.Sanders, ∆ - stepping: a parallelizable shortest path algorithm. Journal of Algorithms 49 (2003)
![Page 39: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/39.jpg)
∆ - stepping algorithm
• Label-correcting algorithm: Can relax edges from unsettled vertices also
• “approximate bucket implementation of Dijkstra”• For random edge weighs [0,1], runs in where L = max distance from source to any node• Vertices are ordered using buckets of width ∆• Each bucket may be processed in parallel• Basic operation: Relax ( e(u,v) )
d(v) = min { d(v), d(u) + w(u, v) }
∆ < min w(e) : Degenerates into Dijkstra∆ > max w(e) : Degenerates into Bellman-Ford
![Page 40: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/40.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket∞ ∞ ∞ ∞ ∞ ∞ ∞
∆ = 0.1 (say)
![Page 41: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/41.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket0 ∞ ∞ ∞ ∞ ∞ ∞
Initialization:Insert s into bucket, d(s) = 000
![Page 42: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/42.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket
0 ∞ ∞ ∞ ∞ ∞ ∞
002R
S
.01
![Page 43: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/43.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket0 ∞ ∞ ∞ ∞ ∞ ∞
2R
0S
.010
![Page 44: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/44.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket0 ∞ .01 ∞ ∞ ∞ ∞
2R
0S
0
![Page 45: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/45.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket0 ∞ .01 ∞ ∞ ∞ ∞
2R
0S
01 3
.03 .06
![Page 46: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/46.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket0 ∞ .01 ∞ ∞ ∞ ∞
R
0S
01 3
.03 .06
2
![Page 47: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/47.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket0 .03 .01 .06 ∞ ∞ ∞
R
0S
0
2
1 3
![Page 48: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/48.jpg)
0.01
∆ - stepping algorithm: illustration
1
2
3
4
5
6
0.13
0
0.18
0.15
0.05
0.07
0.23
0.56
0.02
d array0 1 2 3 4 5 6
Buckets
One parallel phasewhile (bucket is non-empty)
i) Inspect light (w < ∆) edgesii) Construct a set of “requests”
(R)iii) Clear the current bucketiv) Remember deleted vertices
(S)v) Relax request pairs in R
Relax heavy request pairs (from S)Go on to the next bucket0 .03 .01 .06 .16 .29 .62
R
0S
1
2 1 32
6
4
5
6
![Page 49: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/49.jpg)
No. of phases (machine-independent performance count)
Graph Family
Rnd-rnd Rnd-logU Scale-free LGrid-rnd LGrid-logU SqGrid USAd NE USAt NE
No.
of
pha
ses
10
100
1000
10000
100000
1000000
low diameter
high diameter
Too many phases in high diameter graphs: Level-synchronous breadth-first search has the same problem.
![Page 50: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/50.jpg)
Average shortest path weight for various graph families~ 220 vertices, 222 edges, directed graph, edge weights normalized to [0,1]
Graph Family
Rnd-rnd Rnd-logU Scale-free LGrid-rnd LGrid-logU SqGrid USAd NE USAt NE
Ave
rage
sho
rtes
t pa
th w
eigh
t
0.01
0.1
1
10
100
1000
10000
100000Complexity:
L: maximum distance (shortest path weight)
![Page 51: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/51.jpg)
PHAST – hardware accelerated shortest path trees
Preprocessing: Build a contraction hierarchy• order nodes by importance (highway dimension) • process in order • add shortcuts to preserve distances • assign levels (ca. 150 in road networks) • 75% increase in number of edges (for road networks)
D. Delling, A. V. Goldberg, A. Nowatzyk, and R. F. Werneck. PHAST: Hardware- Accelerated Shortest Path Trees. In IPDPS. IEEE Computer Society, 2011
![Page 52: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/52.jpg)
PHAST – hardware accelerated shortest path trees
Order vertices by importance
Eliminate vertex 1
Add shortcut among all higher numbered neighbors of vertex 1
![Page 53: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/53.jpg)
PHAST – hardware accelerated shortest path trees
One-to-all search from source s: • Run forward search from s• Only follow edges to more important nodes• Set distance labels d of reached nodes
![Page 54: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/54.jpg)
PHAST – hardware accelerated shortest path trees
From top-down: • process all nodes u in reverse level order:• check incoming arcs (v,u) with lev(v) > lev(u)• Set d(u)=min{d(u),d(v)+w(v,u)}
![Page 55: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/55.jpg)
PHAST – hardware accelerated shortest path trees
From top-down: • linear sweep without priority queue• reorder nodes, arcs, distance labels by level• accesses to d array becomes contiguous (no jumps)• parallelize with SIMD (SSE instructions, and/or GPU)
![Page 56: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/56.jpg)
PHAST – performance comparison
Edge weights: estimated travel times
Edge weights: physical distances
GPU implementation
Inputs: Europe/USA Road Networks
![Page 57: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/57.jpg)
PHAST – hardware accelerated shortest path trees
• Specifically designed for road networks• Fill-in can be much higher for other graphs
(Hint: think about sparse Gaussian Elimination)
• Road networks are (almost)
planar. • Planar graphs have
separators.• Good separators lead to
orderings with minimal fill.
Lipton, Richard J.; Tarjan, Robert E. (1979), "A separator theorem for planar graphs", SIAM Journal on Applied Mathematics 36 (2): 177–189Alan George. “Nested Dissection of a Regular Finite Element Mesh”. SIAM Journal on Numerical Analysis, Vol. 10, No. 2 (Apr., 1973), 345-363
![Page 58: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/58.jpg)
• Input: Directed graph with “costs” on edges• Find least-cost paths between all reachable vertex pairs• Classical algorithm: Floyd-Warshall
• It turns out a previously overlooked recursive version is more parallelizable than the triple nested loop
for k=1:n // the induction sequencefor i = 1:n for j = 1:n
if(w(i→k)+w(k→j) < w(i→j))w(i→j):= w(i→k) +
w(k→j)
1 52 3 4
1
5
2
3
4
k = 1 case
All-pairs shortest-paths problem
![Page 59: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/59.jpg)
6
3
12
5
43
39
5
7
-3 4
4-1
1
4
5
-2 A B
C DA
B
DC
A = A*; % recursive call
B = AB; C = CA;
D = D + CB;
D = D*; % recursive call
B = BD; C = DC;
A = A + BC;
+ is “min”, × is “add”
V1 V2
![Page 60: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/60.jpg)
6
3
12
5
43
39
5
7
-3 4
4-1
1
4
5
-2
C B
=
The cost of 3-1-2 path
8
Distances Parents
![Page 61: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/61.jpg)
6
3
12
5
43
36
5
7
-3 4
4-1
1
4
5
-28
D = D*: no change
B
=
D Path: 1-2-3
Distances Parents
![Page 62: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/62.jpg)
All-pairs shortest-paths problem
480x
Floyd-Warshall ported to GPU
The right primitive (Matrix multiply)
Naïve recursive implementation
A. Buluç, J. R. Gilbert, and C. Budak. Solving path problems on the GPU. Parallel Computing, 36(5-6):241 - 253, 2010.
![Page 63: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/63.jpg)
Bandwidth:
Latency:
c: number of replicas
Optimal for any memory size !
Communication-avoiding APSP in distributed memory
![Page 64: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/64.jpg)
E. Solomonik, A. Buluç, and J. Demmel. Minimizing communication in all-pairs shortest paths. In Proceedings of the IPDPS. 2013.
Communication-avoiding APSP in distributed memory
![Page 65: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/65.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 66: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/66.jpg)
Betweenness Centrality (BC)
• Centrality: Quantitative measure to capture the importance of a vertex/edge in a graph– degree, closeness, eigenvalue, betweenness
• Betweenness Centrality
( : No. of shortest paths between s and t)• Applied to several real-world networks
– Social interactions– WWW– Epidemiology– Systems biology
tvs st
st vvBC
)()(
st
![Page 67: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/67.jpg)
Algorithms for Computing Betweenness
• All-pairs shortest path approach: compute the length and number of shortest paths between all s-t pairs (O(n3) time), sum up the fractional dependency values (O(n2) space).
• Brandes’ algorithm (2003): Augment a single-source shortest path computation to count paths; uses the Bellman criterion; O(mn) work and O(m+n) space on unweighted graph.
Dependency of source on v.
Number of shortest paths from source to w
![Page 68: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/68.jpg)
Parallel BC algorithms for unweighted graphs
• High-level idea: Level-synchronous parallel breadth-first search augmented to compute centrality scores
• Two steps (both implemented as BFS)– traversal and path counting– dependency accumulation
Shared-memory parallel algorithms for betweenness centrality Exact algorithm: O(mn) work, O(m+n) space, O(nD+nm/p) time.
Improved with lower synchronization overhead and fewer non-contiguous memory references.
K. Madduri, D. Ediger, K. Jiang, D.A. Bader, and D. Chavarria-Miranda. A faster parallel algorithm and efficient multithreaded implementations for evaluating betweenness centrality on massive datasets. MTAAP 2009
D.A. Bader and K. Madduri. Parallel algorithms for evaluating centrality indices in real-world networks, ICPP’08
![Page 69: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/69.jpg)
Parallel BC Algorithm Illustration
0 7
5
3
8
2
4 6
1
9
![Page 70: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/70.jpg)
Parallel BC Algorithm Illustration
1. Traversal step: visit adjacent vertices, update distanceand path counts.
0 7
5
3
8
2
4 6
1
9source vertex
![Page 71: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/71.jpg)
Parallel BC Algorithm Illustration
1. Traversal step: visit adjacent vertices, update distanceand path counts.
0 7
5
3
8
2
4 6
1
9source vertex
2750
0
1
1
1
D
0
0 0
0
S P
![Page 72: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/72.jpg)
Parallel BC Algorithm Illustration
1. Traversal step: visit adjacent vertices, update distanceand path counts.
0 7
5
3
8
2
4 6
1
9source vertex
8
2750
0
12
1
12
D
0
0 0
0
S P
3
2 7
5 7
Level-synchronous approach: The adjacencies of all vertices in the current frontier can be visited in parallel
![Page 73: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/73.jpg)
Parallel BC Algorithm Illustration
1. Traversal step: at the end, we have all reachable vertices,their corresponding predecessor multisets, and D values.
0 7
5
3
8
2
4 6
1
9source vertex
21648
2750
0
12
1
12
D
0
0 0
0
S P
3
2 7
5 7
Level-synchronous approach: The adjacencies of all vertices in the current frontier can be visited in parallel
3 8
8
6
6
![Page 74: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/74.jpg)
Parallel BC Algorithm Illustration
2. Accumulation step: Pop vertices from stack, update dependence scores.
0 7
5
3
8
2
4 6
1
9source vertex
21648
2750
Delta
0
0 0
0
S P
3
2 7
5 7
3 8
8
6
6
)(1)(
)()(
)(
ww
vv
wPv
![Page 75: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/75.jpg)
Parallel BC Algorithm Illustration
2. Accumulation step: Can also be done in a level-synchronous manner.
0 7
5
3
8
2
4 6
1
9source vertex
21648
2750
Delta
0
0 0
0
S P
3
2 7
5 7
3 8
8
6
6
![Page 76: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/76.jpg)
Distributed memory BC algorithm
6
BAT (ATX).*¬B
3
4 7 5
Work-efficient parallel breadth-first search via parallel sparse matrix-matrix multiplication over semirings
Encapsulates three level of parallelism:1. columns(B): multiple BFS searches in parallel2. columns(AT)+rows(B): parallel over frontier vertices in each BFS3. rows(AT): parallel over incident edges of each frontier vertex
A. Buluç, J.R. Gilbert. The Combinatorial BLAS: Design, implementation, and applications. The International Journal of High Performance Computing Applications, 2011.
![Page 77: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/77.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 78: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/78.jpg)
Maximal Independent Set
1
8765
43
2
• Graph with vertices V = {1,2,…,n}
• A set S of vertices is independent if no two vertices in S are neighbors.
• An independent set S is maximal if it is impossible to add another vertex and stay independent
• An independent set S is maximum if no other independent set has more vertices
• Finding a maximum independent set is intractably difficult (NP-hard)
• Finding a maximal independent set is easy, at least on one processor.
The set of red vertices S = {4, 5} is independent
and is maximalbut not maximum
![Page 79: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/79.jpg)
Sequential Maximal Independent Set Algorithm
1
8765
43
21. S = empty set;
2. for vertex v = 1 to n {
3. if (v has no neighbor in S) {
4. add v to S
5. }
6. }
S = { }
![Page 80: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/80.jpg)
1
8765
43
21. S = empty set;
2. for vertex v = 1 to n {
3. if (v has no neighbor in S) {
4. add v to S
5. }
6. }
S = { 1 }
Sequential Maximal Independent Set Algorithm
![Page 81: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/81.jpg)
1
8765
43
21. S = empty set;
2. for vertex v = 1 to n {
3. if (v has no neighbor in S) {
4. add v to S
5. }
6. }
S = { 1, 5 }
Sequential Maximal Independent Set Algorithm
![Page 82: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/82.jpg)
1
8765
43
21. S = empty set;
2. for vertex v = 1 to n {
3. if (v has no neighbor in S) {
4. add v to S
5. }
6. }
S = { 1, 5, 6 }
work ~ O(n), but span ~O(n)
Sequential Maximal Independent Set Algorithm
![Page 83: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/83.jpg)
Parallel, Randomized MIS Algorithm
1
8765
43
21. S = empty set; C = V;
2. while C is not empty {
3. label each v in C with a random r(v);
4. for all v in C in parallel {
5. if r(v) < min( r(neighbors of v) ) {
6. move v from C to S;
7. remove neighbors of v from C;
8. }
9. }
10. }
S = { }
C = { 1, 2, 3, 4, 5, 6, 7, 8 }
M. Luby. "A Simple Parallel Algorithm for the Maximal Independent Set Problem". SIAM Journal on Computing 15 (4): 1036–1053, 1986
![Page 84: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/84.jpg)
1
8765
43
21. S = empty set; C = V;
2. while C is not empty {
3. label each v in C with a random r(v);
4. for all v in C in parallel {
5. if r(v) < min( r(neighbors of v) ) {
6. move v from C to S;
7. remove neighbors of v from C;
8. }
9. }
10. }
S = { }
C = { 1, 2, 3, 4, 5, 6, 7, 8 }
Parallel, Randomized MIS Algorithm
![Page 85: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/85.jpg)
1
8765
43
21. S = empty set; C = V;
2. while C is not empty {
3. label each v in C with a random r(v);
4. for all v in C in parallel {
5. if r(v) < min( r(neighbors of v) ) {
6. move v from C to S;
7. remove neighbors of v from C;
8. }
9. }
10. }
S = { }
C = { 1, 2, 3, 4, 5, 6, 7, 8 }
2.6 4.1
5.9 3.1
1.25.8
9.39.7
Parallel, Randomized MIS Algorithm
![Page 86: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/86.jpg)
1
8765
43
21. S = empty set; C = V;
2. while C is not empty {
3. label each v in C with a random r(v);
4. for all v in C in parallel {
5. if r(v) < min( r(neighbors of v) ) {
6. move v from C to S;
7. remove neighbors of v from C;
8. }
9. }
10. }
S = { 1, 5 }
C = { 6, 8 }
2.6 4.1
5.9 3.1
1.25.8
9.39.7
Parallel, Randomized MIS Algorithm
![Page 87: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/87.jpg)
1
8765
43
21. S = empty set; C = V;
2. while C is not empty {
3. label each v in C with a random r(v);
4. for all v in C in parallel {
5. if r(v) < min( r(neighbors of v) ) {
6. move v from C to S;
7. remove neighbors of v from C;
8. }
9. }
10. }
S = { 1, 5 }
C = { 6, 8 }
2.7
1.8
Parallel, Randomized MIS Algorithm
![Page 88: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/88.jpg)
1
8765
43
21. S = empty set; C = V;
2. while C is not empty {
3. label each v in C with a random r(v);
4. for all v in C in parallel {
5. if r(v) < min( r(neighbors of v) ) {
6. move v from C to S;
7. remove neighbors of v from C;
8. }
9. }
10. }
S = { 1, 5, 8 }
C = { }
2.7
1.8
Parallel, Randomized MIS Algorithm
![Page 89: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/89.jpg)
1
8765
43
21. S = empty set; C = V;
2. while C is not empty {
3. label each v in C with a random r(v);
4. for all v in C in parallel {
5. if r(v) < min( r(neighbors of v) ) {
6. move v from C to S;
7. remove neighbors of v from C;
8. }
9. }
10. }
Theorem: This algorithm “very probably” finishes within O(log n) rounds.
work ~ O(n log n), but span ~O(log n)
Parallel, Randomized MIS Algorithm
![Page 90: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/90.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 91: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/91.jpg)
Strongly connected components (SCC)
• Symmetric permutation to block triangular form
• Find P in linear time by depth-first search
1 52 4 7 3 61
5
2
4
7
3
6
1 2
3
4 7
6
5
Tarjan, R. E. (1972), "Depth-first search and linear graph algorithms", SIAM Journal on Computing 1 (2): 146–160
![Page 92: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/92.jpg)
Strongly connected components of directed graph
• Sequential: use depth-first search (Tarjan); work=O(m+n) for m=|E|, n=|V|.
• DFS seems to be inherently sequential.
• Parallel: divide-and-conquer and BFS (Fleischer et al.); worst-case span O(n) but good in practice on many graphs.
L. Fleischer, B. Hendrickson, and A. Pınar. On identifying strongly connected components in parallel. Parallel and Distributed Processing, pages 505–511, 2000.
![Page 93: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/93.jpg)
Fleischer/Hendrickson/Pinar algorithm
- Partition the given graph into three disjoint subgraphs
- Each can be processed independently/recursively
FW(v): vertices reachable from vertex v. BW(v): vertices from which v is reachable.
Lemma: FW(v)∩ BW(v) is a unique SCC for any v. For every other SCC s, either (a) s ⊂ FW(v)\BW(v),(b) s BW(v)\FW(v),⊂(c) s V \ (FW(v) BW(v)).⊂ ∪
![Page 94: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/94.jpg)
Improving FW/BW with parallel BFS
Observation: Real world graphs have giant SCCs
Finding FW(pivot) and BW(pivot) can dominate the running time with span=O(N)
Solution: Use parallel BFS to limit span to diameter(SCC)
S. Hong, N.C. Rodia, and K. Olukotun. On Fast Parallel Detection of Strongly Connected Components (SCC) in Small-World Graphs. Proc. Supercomputing, 2013
- Remaining SCCs are very small; increasing span of the recursion. + Find weakly-connected components and process them in parallel
![Page 95: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/95.jpg)
• Applications• Designing parallel graph algorithms• Case studies:
A. Graph traversals: Breadth-first searchB. Shortest Paths: Delta-stepping, PHAST, Floyd-WarshallC. Ranking: Betweenness centralityD. Maximal Independent Sets: Luby’s algorithmE. Strongly Connected Components
• Wrap-up: challenges on current systems
Lecture Outline
![Page 96: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/96.jpg)
Problem Size (Log2 # of vertices)
10 12 14 16 18 20 22 24 26
Per
form
ance
rat
e(M
illio
n T
rave
rsed
Edg
es/s
)
0
10
20
30
40
50
60
Serial Performance of “approximate betweenness centrality” on a 2.67 GHz Intel Xeon 5560 (12 GB RAM, 8MB L3 cache)
Input: Synthetic R-MAT graphs (# of edges m = 8n)
The locality challenge“Large memory footprint, low spatial and temporal locality impede performance”
~ 5X drop in performance
No Last-level Cache (LLC) misses
O(m) LLC misses
![Page 97: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/97.jpg)
• Graph topology assumptions in classical algorithms do not match real-world datasets
• Parallelization strategies at loggerheads with techniques for enhancing memory locality
• Classical “work-efficient” graph algorithms may not fully exploit new architectural features– Increasing complexity of memory hierarchy, processor heterogeneity,
wide SIMD.• Tuning implementation to minimize parallel overhead is non-
trivial– Shared memory: minimizing overhead of locks, barriers.– Distributed memory: bounding message buffer sizes, bundling
messages, overlapping communication w/ computation.
The parallel scaling challenge “Classical parallel graph algorithms perform poorly on current parallel systems”
![Page 98: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/98.jpg)
Designing parallel algorithms for large sparse graph analysis
Problem size (n: # of vertices/edges)
104 106 108 1012
Work performed
O(n)
O(n2)O(nlog n)
“RandomAccess”-like
“Stream”-like
Improvelocality
Data reduction/Compression
Faster methods
Peta+
System requirements: High (on-chip memory, DRAM, network, IO) bandwidth.Solution: Efficiently utilize available memory bandwidth.
Algorithmic innovation to avoid corner cases.
Prob
lem
Co
mpl
exity
Locality
![Page 99: CS267, Spring 2014 April 15, 2014 Parallel Graph Algorithms Aydın Buluç ABuluc@lbl.gov aydin/ Lawrence Berkeley National Laboratory](https://reader035.vdocuments.us/reader035/viewer/2022070401/56649f1a5503460f94c2f875/html5/thumbnails/99.jpg)
• Best algorithm depends on the input.• Communication costs (and hence data
distribution) is crucial for distributed memory. • Locality is crucial for in-node performance.• Graph traversal (BFS) is fundamental building
block for more complex algorithms.• Best parallel algorithm can be significantly
different than the best serial algorithm.
Conclusions