Performance Characterization of Multi-threaded Graph ProcessingApplications on Many-Integrated-Core Architecture

Lei Jiang Langshi Chen Judy QiuSchool of Informatics, Computing, and Engineering, Indiana University Bloomington

{jiang60, lc37, xqiu}@indiana.edu

Abstract—In the age of Big Data, parallel graph processing hasbeen a critical technique to analyze and understand connecteddata. Meanwhile, Moore’s Law continues by integrating morecores into a single chip in deep-nano regime. Many-Integrated-Core (MIC) processors emerge as a promising solution to processlarge graphs. In this paper, we empirically evaluate variouscomputing platforms including an Intel Xeon E5 CPU, anNvidia Tesla P40 GPU and a Xeon Phi 7210 MIC processorcodenamed Knights Landing (KNL) in the domain of parallelgraph processing. We show that the KNL gains encouragingperformance and power efficiency when processing graphs, sothat it can become an auspicious alternative to traditionalCPUs and GPUs. We further characterize the impact of KNLarchitectural enhancements on the performance of a state-of-the-art graph framework. We have four key observations:❶ Differentgraph applications require distinctive numbers of threads toreach the peak performance. For the same application, variousdatasets need even different numbers of threads to achieve thebest performance.❷ Not all graph applications actually benefitfrom high bandwidth MCDRAMs, while some of them favorlow latency DDR4 DRAMs. ❸ Vector processing units executingAVX512 SIMD instructions on KNLs are underutilized whenrunning the state-of-the-art graph framework. ❹ The sub-NUMAcache clustering mode offering the lowest local memory accesslatency hurts the performance of graph benchmarks that are lackof NUMA awareness. At last, we suggest future works includingsystem auto-tuning tools and graph framework optimizations tofully exploit the potential of KNL for parallel graph processing.

I. I NTRODUCTION

Big Data explodes exponentially in the form of large-scalegraphs. Graph processing has been an important technique tocompute, analyze and visualize connected data. Real-worldlarge-scale graphs [26], [35] include social networks, trans-portation networks, citation graphs and cognitive networks,which typically have millions of vertices and millions tobillions of edges. Because of the huge graph size, it isnatural for both industry and academia to develop parallelgraph frameworks to accelerate graph processing on varioushardware platforms. A large number of CPU frameworks, e.g.,Giraph [32], Pregel [23], Galois [27], PowerGraph [12] andGraphLab [22], have emerged for scalable in-memory graphprocessing. Recent works propose GPU frameworks such asCuSha [15], Medusa [37], LonestarGPU [5] and Gunrock [35]to perform high throughput graph processing on GPUs.

In CPU frameworks, graphs are partitioned, distributed andprocessed among multiple nodes [12], [22], [23], [32] by mes-sage passing schemes or computed locally on a shared memorynode [26], [27]. Distributed graph processing is notoriousfor frequent communications between computations on each

vertex or edge [12], [26], [27]. In a large-scale cluster, frequentcommunications are translated to a huge volume of messagesacross multiple nodes [22], [23], [32], seriously limitingtheperformance of graph processing. Even a single Mac-MiniSSD-based laptop potentially can outperform a medium-scalecluster for graph processing [17]. On a shared memory multi-core CPU node, communications are interpreted to loads andstores in memory hierarchy [26], [27]. The core efficiency ofa shared memory node is averagely 100× higher than thatin a cluster [30]. However, compared to GPUs, the graphcomputing throughput on CPUs is still constrained by thelimited number of cores.

GPU frameworks capture graph processing parallelism bymapping and computing vertices or edges on thousands oftiny GPU cores [5], [35]. Although graph applications exhibitirregular memory access patterns, frequent thread synchro-nizations and data dependent control flows [28], GPUs stillsubstantially boost the performance of graph processing overCPUs. However, the performance of graph applications onGPUs is sensitive to graph topologies. When traversing graphswith large diameter and small degree, GPUs exhibit poorperformance due to the lack of traversal parallelism [35].Moreover, in some commercial applications, simple opera-tions, such as adding or deleting an edge in a graph, maycost a significant portion of application execution time [26],but it is difficult for GPUs to support such basic operations.

Recently, Intel releases Xeon Phi MIC processors as analternative to traditional CPUs and GPUs for the high per-formance computing market. A MIC processor delivers GPU-comparable computing throughput and CPU-like sequentialexecution power. Compared to GPUs, a MIC processor caneasily support basic graph operations, e.g., inserting/deleting anode/edge, since it is fully compatible with the X86 instructionset. In this paper, we focus on the performance characterizationof multi-threaded graph applications on an Intel Xeon Phi(2nd generation) processor - KNL [29]. Our contributions aresummarized as follows:

• We empirically evaluate a state-of-the-art graph frame-work on three hardware platforms: an Intel Xeon E5 CPU,an Nvidia Tesla P40 GPU and a KNL MIC processor.We show that the KNL demonstrates encouraging perfor-mance and power efficiency when running multi-threadedgraph applications.

• We further characterize performance details of multi-threaded graph benchmarks on KNL. We measure and an-alyze the impact of KNL architectural enhancements such

as many out-of-order cores, simultaneous multithreading,cache clustering modes, vectorization processing unitsand 3D stacked MCDRAM on parallel graph processing.We have four key observations:❶ Different graph appli-cations require distinctive numbers of threads to achievetheir best performance. For the same benchmark, variousdatasets ask for different numbers of threads to attainthe shortest execution time.❷ Not all graph applicationsactually benefit from high bandwidth MCDRAMs, whilesome of them favor low latency DDR4 DRAMs.❸VPUs executing AVX512 SIMD instructions on KNLsare underutilized when processing graphs.❹ The sub-NUMA cache clustering mode offering the lowest localmemory access latency hurts the performance of graphbenchmarks that are lack of NUMA awareness.

• For future work, we suggest possible system auto-tuningtools and framework optimizations to fully exploit thepotential of KNL for multi-threaded graph processing.

II. BACKGROUND

A. Graph Processing Frameworks and Benchmark Suite

The rapid and wide deployment of graph analytics in real-world diversifies graph applications. A lot of applicationssuchas breadth first search incorporate graph traversals, whileotherbenchmarks such as page rank involve intensive computationson vertex properties. Though low-level hardwired implementa-tions [1], [4], [8], [10], [13] have demonstrated high computingefficiency on both CPUs and GPUs, programmers need high-level programmable frameworks to implement a wide vari-ety of complex graph applications to solve real-world prob-lems. Therefore, graph processing heavily depends on specificframeworks composed of data structures, programming modelsand basic graph primitives to fulfill various functionalities.Previous research efforts propose many graph frameworks onboth CPUs [12], [22], [26], [27] and GPUs [5], [15], [26], [37]to hide complicated details of operating on graphs and providebasic graph primitives. Most workloads spend50% ∼ 76% ofexecution time within graph frameworks [26].

The graph applications we studied in this paper are from astate-of-the-art graph benchmark suite,graphBIG [26]. Thesuite is implemented with IBMSystem-Gframework thatis a comprehensive graph library used by many industrialsolutions [31]. The framework adopts the vertex centric pro-gramming model, where a vertex is a basic element of a graph.Properties and outgoing edges of a vertex are attached to thesame vertex data structure. The data structure of all vertices isstored in an adjacency list, and outgoing edges inside a vertexdata structure also form an adjacency list of edges. Themajorreason to choose graphBIG is that benchmarks in this suitehave a CPU OpenMP version and a GPU CUDA version shar-ing the same data structure and vertex centric programmingmodel provided by System-G. In this way, we can minimizethe impact of differences between CPU and GPU graphprocessing frameworks, and focus on only differences betweenvarious hardware computing platforms. On the contrary, othergraph suites [1], [4], [5] implement graph applications without

using a general framework or by using different frameworkson CPUs and GPUs respectively. Particularly, as one of thebest-known graph benchmark suites, Graph500 [25] providesonly limited number of CPU workloads, since it is designedfor only CPU-based system performance ranking.

B. Target Graph Benchmarks

Graph applications exhibit diversified program behaviors.Some workloads, e.g., breadth first search, are traversal-based.Only a subset of vertices is typically active at a given pointduring the execution of this type of applications. They oftenintroduce many memory accesses but limited arithmetic oper-ations. Their irregular memory behavior results in extremelypoor locality in memory hierarchy. Some graph benchmarksoperating on rich vertex properties, e.g., page rank and trianglecount. They incorporate heavy arithmetic computations onvertex properties and intensive memory accesses leading tohybrid workload behaviors. Most vertices are active in allstages of their executions. We selected, compared and studiedsix common graph benchmarks in this paper. We introducetheir details as follows:

• Breadth First Search traverses a graph by starting froma root vertex and exploring neighboring nodes first. Thetraversal parallelism is exploited by vertex capture whereeach thread picks a vertex and searches its neighbors.

• K-Core finds a maximal connected sub-graph where allvertices have≥ K degree. It follows Matula & Beck’salgorithm [26].

• Single Source Shortest Pathcalculates the minimumcost paths from a root vertex to each vertex in a givengraph by Dijkstra’s algorithm [26].

• Graph Coloring partitions a graph into independent sets.One set contains vertices sharing the same color. It adoptsLuby-Jones’ algorithm [26].

• Page Rankuses the probability distribution to computepage rankings. The rank of each vertex is computed byPR(u) =

∑Nu

PR(v)E(v) , where thePR of a vertex (u) is

decided by thePR of each vertex (v) in its neighbor set(Nu) divided by its outgoing edge numberE(v).

• Triangle Count measures the number of triangles that areformed in a graph when three vertices are connected toeach other. It is implemented by Schank’s algorithm [26].

C. Graph Topology

The performance of graph applications is heavily influencedby the topology of graph datasets. We studied how topologiesimpact the benchmark performance via following metrics.The eccentricity ǫ(v) of a vertex v in a given connectedgraph G is the maximum graph distance betweenv andany other vertexu of G. The diameter d of a graph isthe maximum eccentricity of any vertex in the graph (d =maxv∈V ǫ(v)). TheFiedler eigenvalueof the graph (F (G)) isthe second smallest eigenvalue of the Laplacian matrix ofG.The magnitude ofF (G) exhibits how wellG is connected.For traversal-based applications such as breadth first search,traversal iteration number is proportional to the eccentricity

and Fiedler eigenvalue. Thevertex degreeindicates the numberof edges connected to a vertex. The average vertex degree andthe vertex degree distribution of a graph reflect the amount ofparallelism. Large vertex degree variations introduce substan-tial load imbalance during graph traversals. Real-world graphscan be categorized into two types: the first has large diameterwith evenly distributed degree, e.g., road networks; and thesecond includes small eccentricity graphs with a subset offew extremely high-degree vertices, e.g., social networks. Wechose different datasets from both categories, and generatedseveral synthesized graphs with diameters varying from smallto huge in Section IV.

D. Intel Xeon Phi Architecture

1) Overall Architecture:Xeon Phi MIC processors are fullycompatible with the x86 instruction set and hence supportstandard parallel shared memory programming tools, modelsand libraries such as OpenMP. The KNL [29] is the 2ndgeneration architecture fabricated by the 14nm technologyandhas been adopted by several data centers such as the nationalenergy research scientific computing center [3].

core

miscEDC

PCIe

dd

r4 mcTile

mcdram

mcdram

mcdram

mcdram2D Mesh

2 VPU

CHA

1MB

L2core

2 VPU

Tile structure

Fig. 1. Xeon Phi KNL (MC: DDR4 controller; EDC: MCDRAM controller).

The KNL detailed architecture is shown in Figure 1. It isbuilt by up to 72 Atom (Silvermont) cores, each of which isbased on a low operating frequency 2-wide issued out-of-order(OoO) micro-architecture supporting four concurrent threads,a.k.a, simultaneous multithreading (SMT). Additionally,everycore has two vector processing units (VPUs) that supportSIMD instructions such asAVX2 and AVX512 that is a new512-bit advanced vector extension of SIMD instructions forthe x86 instruction set. A VPU can execute up to 16 singleprecision operations or 8 double precision floating point op-erations in each cycle. Two cores form a tile sharing a 1MB16-way L2 cache and a caching home agent (CHA), which isa distributed tag directory for cache coherence. All tiles areconnected by a 2D Mesh network-on-chip (NoC).

ddr4mcdramddr4

mcdram ph

ysica

l

ad

dre

ss

flatcache

ddr4

mcdram ph

ysica

l

ad

dre

ss

hybrid

mcdram

Fig. 2. Configurations on MCDRAM.

2) MCDRAM: The KNL main memory system supportsup to 384GB of DDR4 DRAM and 8∼16GB of 3D stackedMCDRAM. The MCDRAM significantly boosts the memorybandwidth, but the access latency of MCDRAM is actuallylonger than that of DDR4 DRAM [24]. As Figure 2 shows,MCDRAM can be configured as a parallel memory component

to DDR4 DRAM in main memory (flat mode), a hardware-managed L3 cache (cache mode) or both (hybrid mode).The flat mode offers high bandwidth by MCDRAM and lowlatency by DDR4 DRAM. However, programmers have totrack the location of each data element and manage softwarecomplexity. On the contrary, thecachemode manages MC-DRAM as a software-transparent direct-mapped cache. Thehybrid mode combines the other two modes.

All-to-All

❶

❸

Quadrant(Hemisphere) SNC-4(2)

❶❷

❸

❶

❸❷

❹ ❹❷

❹

Fig. 3. Configurations on Cache Clustring.

3) Cache Clustering Mode:Since all KNL tiles are con-nected by a Mesh NoC where each vertical/horizontal link isa bidirectional ring, all L2 caches are maintained coherentbythe MESIF protocol. To enforce cache coherency, KNL has adistributed cache tag directory organized as a set of per-tile tagdirectories (shown as CHA in Figure 1) that record the stateand location of all memory lines. For any memory address, bya hash function, KNL identifies which tag directory recordingit. As Figure 3 shows, the KNL cache can be operated infive modes includingAll-to-All, Hemisphere, Quadrant, SNC-2 andSNC-4. When❶ a core confronts a L2 miss,❷ it sendsa request to look up a tag directory.❸ The directory finds amiss and transfers this request to a memory controller.❹ Atlast, the memory controller fetches data from main memoryand returns it to the core enduring the L2 miss. In theAll-to-All mode, memory addresses are uniformly hashed acrossall tag directories. During a L2 miss, the core may send arequest to a tag directory physically located in the farthestquadrant from the core. After the directory finds a miss, itmay also send this request to a memory controller located ina third quadrant. Therefore, a L2 miss may have to traversethe entire Mesh to read one line from main memory. TheQuadrant(Hemisphere) mode divides a KNL chip into four(two) parts. During a L2 miss, the core still needs to send arequest to every on-chip tag directory, but the data associatedto the target tag directory must be in the same part that thetag directory is located. Memory accesses from a tag directoryare managed by its local memory or cache controller. TheHemisphereand Quadrant modes are managed by hardwareand transparent to Operating System (OS). In contrast, the sub-NUMA-clustering (SNC-2/4) also separates the chip into twoor four parts and exposes each as a separate NUMA domainto OS. During a L2 miss, the core only needs to send a requestto its local tag directory that also transfers this request to thelocal memory controller if there is a directory miss. Therefore,the SNC-4mode has the lowest local memory access latency,but longer memory latency than that of theQuadrant modewhen a request crosses NUMA boundaries. This is becausethe requests traveling to another NUMA region have to bemanaged by OS or applications themselves.

III. R ELATED WORK

A recent graph characterization work on CPUs [4] identifiesthat eight fat OoO Ivy Bridge cores fail to fully utilize off-chip memory bandwidth when processing graphs, since thesmall instruction window cannot produce enough outstandingmemory requests. In contrast, the KNL MIC processor can sat-urate main memory bandwidth by> 64 small OoO cores, eachof which supports SMT and has two VPUs. More researchefforts [2], [5], [28], [36] analyze GPU micro-architectural de-tails to identify bottlenecks of graph benchmarks. In this paper,we analyze and pinpoint inefficiencies on micro-architecturesof KNL when running multi-threaded graph applications. Pre-vious physical-machine-based works [7], [13] find that the firstgeneration Xeon Phi, Knight Corner, has poor performancefor graph applications due to its feeble cores. Although thelatest simulator-based work [1] characterizes the performanceof multi-threaded graph applications on a MIC processorcomposed of 256 single-issue in-order cores, it fails to considerthe architectural enhancements offered by KNLs, e.g., OoOcores, SMT, cache clustering modes, VPUs and MCDRAM.To our best knowledge, this work is the first to characterize andanalyze the performance of multi-threaded graph applicationson the KNL MIC architecture.

IV. EXPERIMENTAL METHODOLOGY

Graph benchmarks. In this paper, we adopted and studiedsix benchmarks including breadth first search (BF), singlesource shortest path (SS), graph coloring (GC), k-core (KC),triangle count (TC) and page rank (PR) from graphBIG [26].Each benchmark contains a CPU OpenMP version and a GPUCUDA version. Due to the absence of PR GPU code in graph-BIG, we created a vertex-centric GPU PR implementationbased on a previous GPU PR benchmark [5] and the System-G framework. We compiled codes on CPU by icc (17.0.0)with -O3, -MIC-AVX-512 and Intel OpenMP library. Throughthe commandnumactl, we chose to run CPU benchmarks inDDR4 DRAM under the flat mode. On GPU, we compiledprograms by nvcc (V8.0.61) with CUDA-8.0. Since the sizeof large graph datasets exceeds the maximum physical GPUmemory capacity, we usedcudaMallocManagedfunction toallocate data structures into a CUDA-managed memory spacewhere both CPUs and GPUs share a single coherent addressspace. In this way, CUDA libraries transparently manage GPUmemory access, data locality and data migrations betweenCPU and GPU. The OS we used is Ubuntu-16.04.

TABLE IBENCHMARK DATASETS.

Dataset Abbr. Vertices Edges Degree IterBF

roadNet_CAroad

1.97M 5.53M 2.81 665roadNet_TX 1.38M 3.84M 2.27 739soc-Slashdot0811

social0.08M 0.9M 11.7 18

ego-Gplus 0.11M 13.6M 127.1 6delaunay_n18

delaunay0.26M 1.57M 6.0 223

delaunay_n19 0.52M 3.15M 6.0 296kron_g500-logn17

kron0.11M 10.23M 78.0 3

kron_g500-logn18 0.21M 21.17M 80.7 3twitter7

large17M 0.48B 4.6 23

com-friendster 65.6M 1.81B 27.6 15

Graph datasets. The graph inputs for our simulated bench-marks are summarized in Table I, whereDegree denotes theaverage vertex degree of the graph andIterBF describes theaverage BF iteration number computed by searching from 10Krandom root vertices.roadNet_XX(road) are real-world roadnetworks where most vertices have an outgoing degree below4. soc-Slashdot0811and ego-Gplus(social) are two socialnetworks typically having scale-free vertex degree distributionand small diameter.delaunay(delaunay) datasets are Delau-nay triangulations of random points in the plane that alsohave extremely small outgoing degrees. Like social networks,kronecker(kron) datasets have large average vertex outgoingdegree and can be traversed by only several BF iterations. Wealso included two large graph datasets (large),twitter7 andcom-friendster, each of which contains millions of verticesand billions of edges. The size of graph datasets is mainlydecided by the number of edges, because the average degreeof graphs we chose is≥ 2. We selected graph datasets fromthe Stanford SNAP [19]. We also used the synthetic graphgenerator, PaRMAT [14], in dataset sensitivity studies.

TABLE IIMACHINE CONFIGURATIONS.

Hardware Description

Intel Xeon launched in Q2’16, 8-core@2.5GHz, 2T/core,E5-4655v4 3.75MB L3/core, 512GB 68GB/s DDR4 DRAM

thermal design power (TDP) 135WNvidia launched in Q2’16, 1920-cudaCore@1.5GHz,GTX1070 peak SP perf: 5783 GFLOPS, 8GB 256GB/s

GDDR5, PCIe 3.0×16 to CPU, TDP 150WNvidia Tesla launched in Q3’16, 3840-cudaCore@1.3GHz,P40 peak SP perf: 10007 GFLOPS, 24GB 346GB/s

GDDR5, PCIe 3.0×16 to CPU, TDP 250Wlaunched in Q2’16, 64-core@1.3GHz, 4T/core,

Intel Xeon Phi 0.5MB L2/core, peak SP perf: 5324 GFLOPS,7210 (KNL) 8 channels 16GB 400GB/s 3D MCDRAM,

512GB 102GB/s DDR4 DRAM, TDP 230WDisk 4TB SSD NAND-Flash

Hardware platforms . We chose and compared three server-level hardware platforms including a Xeon E5 CPU, a TeslaP40 GPU and a Xeon Phi 7210 (KNL) MIC processor. Since,compared to KNL, Tesla P40 achieves almost double peaksingle point float (SPF) throughput, we also included anNvidia GTX1070 GPU having similar peak SPF throughputin our experiments. The machine configurations are shown inTable II. Both Xeon E5 and KNL have a 512GB DDR4 mainmemory system, while P40 and GTX1070 are also hosted inthe Xeon E5 machine with 512GB DDR4 DRAM. The thermaldesign power (TDP) values of Xeon E5, GFX1040, P40 andKNL are 135W, 150W, 250W and 230W. TDP represents themaximum amount of heat generated by a hardware componentthat the cooling system can dissipate under any workload.

Profiling tools. We adopted Intel VTune Analyzer to collectarchitectural statistics and likwid-powermeter [34] to profilepower consumption on CPU and KNL. The likwid-powermeteris a tool for accessing RAPL registers on Intel processors, soit is able to measure the power consumption of both CPUpackage and DRAM memories. The register reading intervalwe set in all experiments is1ms. For GPUs, we used Nvidia

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

GCKCSS GFX1070 P40 KNL

BF

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

103

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

103

104

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

PRTC

road

delauna

ysoc

ialkronlarge

gmean

10-1

100

101

102

road

delauna

ysoc

ialkronlarge

gmean

123456789

10no

rmal

ized

TEPS

Fig. 4. Performance comparison between Xeon E5-4655v4, GeForce GTX1070, Tesla P40 and KNL (normalized to Xeon E5-4655v4).

visual profiler (nvprof) to collect both performance and powercharacterization results. The performance results we reportedin Section V do not include the overhead of profiling tools onCPU, GPU and KNL.

Metrics. We measured the performance of graph applica-tions asTraversed Edges Per Second(TEPS) [25]. The TEPSfor non-traversing-based benchmarks, e.g., PR, is computedby dividing the number of processed edges by the meantime per iteration, since each vertex processes all its edgesin one iteration [25]. We aim to understand the performanceand power consumption of multi-threaded graph applicationkernels, and thus all results ignore disk-to-memory and host-to-device data transfer time during application initializations.But we measured page faults on CPU and data transfersbetween CPU and GPU inside application kernels.

V. EVALUATION

We cannot answer which type of hardware is the best forshared memory parallel graph processing, since the platformswe selected have completely different power budgets and hard-ware configurations such as operating frequencies, core num-bers, cache sizes and micro-architecture details. By allocatingmore power budgets and hardware resources, theoreticallyspeaking, any type of platform can possibly outperform theothers. Our purpose of the comparison between KNL and othertypes of hardware is to demonstrate the KNL MIC processorachieves encouraging performance and power efficiency, andthus it can become one of the most promising alternatives totraditional CPUs and GPUs when processing graphs.

To investigate the KNL performance of parallel graph pro-cessing, we empirically analyze the performance comparisonbetween four hardware platforms. And then, we characterizeperformance details of six graph benchmarks on KNL. Weexplain the impact of KNL architectural innovations suchas many OoO cores, SMT, VPU and MCDRAM on multi-threaded graph applications.

A. Performance Comparison

The performance comparison between four hardware plat-forms is shown in Figure 4, where all results are normalizedto the performance (TEPS) of Xeon E5 CPU. We configuredKNL cache clustering in the Quadrant mode and MCDRAM inthe cache mode. The KNL performance reported in this sectionis achieved by its optimal thread configuration including threadnumber and thread affinity. We also performed an exhaustive

search on all possible configurations on CPU and GPU toattain and report the best performance.

The primary weakness of GPU is the load imbalanceproblem introduced by its sensitivity to graph topologies intraversal-based graph applications. For BF, GPUs achieve bet-ter performance than KNL on graphs with small diameter andlarge average vertex degree, e.g., social and kron. Huge traver-sal parallelism exists in these graphs, and hence, the morecores one platform has, the better performance it can achieve.In this case, KNL is slower than GTX1070, although theyhave similar peak SFP computing throughput. This is becausegraph traversal operations can barely take advantage of VPUson KNL. However, compared to KNL, GPUs processes lessedges when traversing graphs with large diameter and smallaverage vertex degree, e.g., road and delaunay. There is littletraversal parallelism in these graphs, so the OoO cores of KNLprevail due to their powerful sequential execution capability.When processing large graphs, GPUs suffer from frequent datamigrations between CPU and GPU memories, due to theirlimited GDDR5 memory capacity. Therefore, KNL shines onthese large graph datasets. P40 outperforms GTX1070 on allgraph inputs, because it has more CUDA cores and a largercapacity GDDR5 memory system. As another traversal-basedgraph benchmark, SS shares the same performance trend asthat of BF.

KC and GC are two applications operating on graph struc-tures. Their computations involve a huge volume of basicarithmetic operations, e.g., integer incrementing (decrement-ing), on vertices or edges. A large number of CUDA coreson GPUs support these massive arithmetic operations betterthan a small number of KNL cores. GPUs can perform fine-grained scheduling on warps to fully utilize CUDA cores,while each KNL core can switch between only four threads,each of which cannot be fully vectorized. Therefore, the GPUperformance is generally 10× better than that of KNL. ForGC, the improvement brought by thousands of CUDA cores onP40 even mitigates the data transfer penalty due to its limitedGDDR5 memory capacity, i.e., P40 improves the processingperformance over KNL by 16.7% on large graph datasets.

As graph benchmarks computing on vertex properties,TC and PR exhibit hybrid workload behaviors, since theyrequire not only graph traversals but also relatively heavyarithmetic operations on vertex properties, e.g., multiply-accumulate operations. In TC, arithmetic operations dominatethe performance of graph processing, so compared to KNL,GPUs boost the application performance by 112%∼145%

CPUC-D

RAMGFX

1070

G-DRAM P40

P-DRAM KNL

K-DRAM

0

50

100

150

200

250Po

wer

(Wat

t)

CPUC-D

RAMGFX

1070

G-DRAM P40

P-DRAM KNL

K-DRAM

0

50

100

150

200

250BF_largeBF_roadBF_social

CPUC-D

RAMGFX

1070

G-DRAM P40

P-DRAM KNL

K-DRAM

0

50

100

150

200

250

CPUC-D

RAMGFX

1070

G-DRAM P40

P-DRAM KNL

K-DRAM

0

50

100

150

200

250GC_large

Fig. 5. Power comparison between Xeon E5-4655v4, GeForce GTX1070, Tesla P40 and KNL.

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

GCKCSS GFX1070 P40 KNL

BF

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

103

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

103

104

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

PRTC

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

road

delauna

ysoc

ialkronlarge

gmean

100

101

102

norm

alize

d TE

PS p

er w

att

Fig. 6. Performance per watt comparison between Xeon E5-4655v4, GeForce GTX1070, Tesla P40 and KNL (normalized to Xeon E5-4655v4).

averagely. In PR, graph traversals dominate on graphs withlarge diameter and small average vertex degree, e.g., road anddelaunay, so KNL obtains better edge processing speed onthese graphs. For almost all combinations of graph benchmarksand datesets, Xeon E5 achieves the worst performance, i.e.,only 1%∼33.9% of CPU performance or 8%∼50% of KNLperformance. Only for TC, GPUs are slightly slower thanXeon E5 on large graph datasets, due to large penalties of datatransfers on PCIe links between CPU and GPU memories.

B. Power and Performance Per Watt Comparison

The details of power characterization is shown in Figure 5,where we list power consumptions of four types of hardwareand their 512GB DRAM main memories represented byX-DRAM (X can be Xeon E5 {C}PU, {G}FX1070, {P}40 and{K}NL.). The KNL power includes the power of MCDRAM,while the GPU power contains the power of its GDDR5memory. For GPUs, we used nvprof to profile the GPU powerand likwid-powermeter to measure the DRAM power in itshost machine.

For social graph datasets, all platforms are able to fullyoccupy almost 100% hardware resources and approach theirTDP in a short time, because large traverse parallelism exists inthese datasets. Only CPU DRAM is heavily accessed duringBFs, while other platforms barely access their 512GB mainmemories. This is because both GPUs has their own GPUmemories and KNL has a 16GB MCDRAM-based cache.As a result, only CPU DRAM achieves> 60W averagepower consumption. For road graph datasets, the largest powerspent by P40 (GFX1070) during searches is only around 48%(46%) of its TDP, due to the lack of traverse parallelismin these datasets. The average power consumption of twoGPUs when processing road datasets decreases by 16%∼24%over that dissipated during traversing on social datasets.Incontrast, compared to social graphs, CPU and KNL slightlydecrease their average power consumption by only 6%∼9%,

and the power consumption of CPU DRAM decreases by10% averagely when processing road graphs. Compared toGPUs, CPU and KNL are insensitive to topologies of graphdatasets due to their limited number of cores. Therefore, theyhave more consistent power results throughout various graphinputs. For large graph datasets, GPUs barely approach theirTDPs due to frequent data transfers through PCIe links. CPUand KNL also suffers from frequent page faults, and thustheir largest power values are only around 70%∼80% of theirTDPs. However, compared to other datasets, DRAMs in allhardware platforms significantly boost the power consumptionby 39%∼62% when dealing with large graph inputs. SSshares a similar power consumption trend with BF. The powerconsumption of TC, KC, GC and PR with small datasets issimilar to BF with social datasets, since all hardware platformsenjoy similarly large graph processing parallelism and smalldatasets do not need to frequently visit DRAMs. But whenrunning TC, KC, GC and PR with large datasets, DRAMsspend 4%∼11% more power than that when processing largedatasets in BF. This is because graph applications operatingon both structures of large graphs and a huge volume of vertexproperties send more intensive memory accesses to DRAMsthan traversal-based benchmarks.

The performance per watt comparison among all hardwareplatforms is exhibited in Figure 6, where all results arenormalized to the performance per watt result of Xeon E5CPU. Compared to CPU, P40 (GFX1070) achieves 2.5∼69×(3∼37×) better performance per watt averagely. On average,KNL obtains only 60%∼285% better performance per wattover CPU. But for large graph datasets, compared to GPUs,KNL improves the performance per watt by 74%∼186% whenrunning BF and PR, and has similar results on performance perwatt when running other benchmarks. Frequent data transfersbetween GPU and CPU memories caused by the limitedcapacity of GDDR5 memory waste a large amount of powerand slow down applications on GPUs when processing large

graphs. When running GC, TC and PR, although P40 is fasterthan GFX1070, the power consumption of P40 is also larger.Therefore, the performance per watt of P40 is not significantlybetter than that of GFX1070 for these benchmarks.

C. Threading1) Thread Scaling:We show the graph application per-

formance with varying OpenMP thread numbers on KNL inFigure 7, where all performance results are normalized to theperformance achieved by 256 threads. 256 (4 threads× 64cores) is the KNL logic core number. Due to the limitedfigure space, we selected only the larger dataset in each graphinput category. With a small number of threads (< 32), thereare not enough working threads to do the task, and thus nobenchmark obtains its best performance. When having morethreads, the performance of almost all benchmarks improvesmore or less. However, when the thread number goes beyond256, the execution time significantly rises, since the threadsynchronization overhead dominates and degrades the perfor-mance of most benchmarks. The best performance of eachbenchmark is typically achieved by32 ∼ 512 threads.

SMT and oversubscription allow multiple independentthreads of execution on a core to better utilize hardwareresources on that core. To implement SMT, some hardwaresections of the core (but not the main execution pipeline) areduplicated to store architectural states. When one thread isstalled by long latency memory accesses, SMT stores its stateto backup hardware sections and switches the core to executeanother thread. SMT transforms one physical KNL core tofour logic cores, each of which supports one thread. Someapplications with certain datasets, e.g., TC with ego-Gplusand SS with kron_g500-logn18, fulfill their best performanceby SMT (256 threads). In contrast, oversubscription requiresthe assistance from software such as OS or OpenMP libraryto switch threads, when the running thread is stalled. Forapplications suffering from massive concurrent cache misses,like SS with roadNet_TX and delaunay_n19, oversubscriptionsupports more simultaneous threads and outruns SMT. Whenrunning these applications, compared to the penalty of longlatency memory accesses, the OS context switching overheadis not significant.

2) The Optimal Thread Number:We define the optimalthread number as the thread number achieving the best per-formance for each benchmark. Figure 8 describes the optimalthread number for all applications with all datasets. Thereisno universal optimal thread number, e.g., the physical corenumber or the logic core number, that can always have the bestperformance for all applications with all datasets. Differentapplications require distinctive optimal thread numbers fortheir own best performance. Moreover, even for the sameapplication, the optimal thread numbers for various graphdatasets are different. If statically setting the thread numberto 256 for all application, Figure 7 shows KC with twitter7 isdecelerated by4.1×.

3) Thread Placement and Affinity:Because graphBIG de-pends on the OpenMP library, we can configure the threadplacement and affinity byKMP_AFFINITY=X, granularity=Y.

Here, X indicates thread placement and has two options:assigning threadn+1 to an available thread context as close aspossible to that of threadn (Compact) or distributing threadsas evenly as possible across the entire system (Scatter). AndY denotes granularity and includes two choices: allowing allthreads bound to a physical core to float between differentthread contexts (Core) or causing each thread to be bound to asingle thread context (Thread). The performance comparisonbetween all configurations of thread placement and affinityis shown in Figure 9, where each bar represents oneX-Ycombination and all bars are normalized to Compact-Core. Wesee that Compact configurations with Core and Thread haveworse performance, since Scatter configurations better utilizeall physical cores and distribute memory requests evenlyamong all memory controllers. In two Scatter configurations,granularity Thread wins slightly better performance, becauseit scatters consecutive threads sharing similar application be-haviors to different physical cores. SMT works better whenthreads with different program behaviors run on the samephysical core. When one thread is stalled by memory accesses,the core is switched to other threads that unlikely confrontmemory stalls in near future due to their distinctive behaviors.

D. MCDRAM

We configured MCDRAMs as a hardware-managed L3cache for KNL as default. However, we can also disableMCDRAMs to use only DDR4 DRAM as main memory.The performance improvement of MCDRAM is exhibitedin Figure 10, where all results are normalized to the per-formance of KNL with only DDR4 DRAM main mem-ory. Compared to DDR4, MCDRAM can supply4× band-width. The MCDRAM-based cache boosts the performanceof most benchmarks by its 16GB capacity and larger band-width. Particularly, large graph datasets benefit more fromtheMCDRAM-based cache, since they enlarge the working setsize for most applications. On the contrary, the access latencyof MCDRAM is longer than that of DDR4 DRAM [24].Some benchmarks processing graphs with large diameter andsmall average degree, e.g. BF with road datasets, are moresensitive to the prolonged memory access latency and actuallydecelerated by the MCDRAM-based cache, because they havea limited number of concurrently pending memory accessesand small memory footprints.

E. Vectorization

We compiled graph programs with icc -O3 with various vec-torization choices: no vectorization,AVX2 and AVX512. Theperformance of graph benchmarks with different vectorizationoptions is shown in Figure 11, where all results are normalizedto the no vectorization schemeNOVEC. AVX2 improves thegraph processing performance by 47%∼324% overNOVEC.However, compared toAVX2, AVX512does not significantlyfurther boost the performance of graph applications. This isbecause the implementation of System G does not explicitlyrepresent vertices or edges by floating point or integer arrays.Instead, vertices and edges are encapsulated into lists or maps

32 64 128 256 512 1024

0

1

2

32 64 128 256 512 1024

0

1

232 64 128 256 512 102

40

1

2

32 64 128 256 512 1024

0

1

232 64 128 256 512 102

40

1

2

32 64 128 256 512 1024

0

1

2

norm

alized

norm

alized

roadNet_TX ego-Gplus delaunay_n19 kron_g500-logn18 Twitterpe

rform

ance

KC

TCGC

thread #thread #

SSperfo

rman

ce

PR

thread #

BF

Fig. 7. The KNL performance with varying thread numbers (normalized to the performance achieved by 256 threads).

road

Net

_CA

road

Net

_TX

soc-

Slas

hdot

0811

ego-

Gpl

usde

laun

ay_n

18de

laun

ay_n

19kr

on_g

500-

logn

17kr

on_g

500-

logn

18tw

itter

com

-frie

ndst

er

road

Net

_CA

road

Net

_TX

soc-

Slas

hdot

0811

ego-

Gpl

usde

laun

ay_n

18de

laun

ay_n

19kr

on_g

500-

logn

17kr

on_g

500-

logn

18tw

itter

com

-frie

ndst

er

road

Net

_CA

road

Net

_TX

soc-

Slas

hdot

0811

ego-

Gpl

usde

laun

ay_n

18de

laun

ay_n

19kr

on_g

500-

logn

17kr

on_g

500-

logn

18tw

itter

com

-frie

ndst

er

road

Net

_CA

road

Net

_TX

soc-

Slas

hdot

0811

ego-

Gpl

usde

laun

ay_n

18de

laun

ay_n

19kr

on_g

500-

logn

17kr

on_g

500-

logn

18tw

itter

com

-frie

ndst

er

road

Net

_CA

road

Net

_TX

soc-

Slas

hdot

0811

ego-

Gpl

usde

laun

ay_n

18de

laun

ay_n

19kr

on_g

500-

logn

17kr

on_g

500-

logn

18tw

itter

com

-frie

ndst

er

road

Net

_CA

road

Net

_TX

soc-

Slas

hdot

0811

ego-

Gpl

usde

laun

ay_n

18de

laun

ay_n

19kr

on_g

500-

logn

17kr

on_g

500-

logn

18tw

itter

com

-frie

ndst

er 8

163264

128256512

1024

PR

TC

GCKCBF

the

optim

al th

read

#

SS

Fig. 8. The optimal thread number for all benchmarks with all datasets

BF SS KC GC TC PR1.01.21.41.61.82.0

norm

alize

d pe

rf.

Compact-Thread Scatter-Core Scatter-Thread

Fig. 9. Performance comparison between various configurations of threadplacement and affinity (normalized to Compact-Core).

BF-road

BF-social BF-la

rge SS KC GC TC PR0.81.01.21.41.6

norm

alize

d pe

rf.

Fig. 10. Performance comparison between DDR4 and MCDRAM (normalizedto DDR4 DRAM main memory without MCDRAM).

in graph frameworks, and thus it is difficult to vectorize thesedata structure on KNL. Moreover, most graph datasets we usedare sparse, so they can barely be improved by wideAVX512SIMD instructions.

BF SS KC GC TC PR1.01.52.02.54.04.5

norm

alize

d pe

rf. AVX2 AVX512

Fig. 11. Performance comparison between AVX2 and AVX512 (normalizedto no vectorization (NOVEC).

BF SS KC GC TC PR0.50.60.70.80.91.0

norm

alize

d pe

rf.

All-to-All Hemisphere SNC2 SNC4

Fig. 12. Performance comparison between different cache clustering modes(normalized toQuadrant).

F. Cache Clustering Mode

The performance comparison between various cache clus-tering modes is shown in Figure 12, where all results arenormalized to Quadrant. As explained in Section II-D3, amongall hardware-managed modes includingAll-to-All, HemisphereandQuadrant, Quadrant achieves the best performance, sinceit can keep both L2 accesses and memory accesses servedwithin the local quadrant on KNL. Two software-managedmodes (SNC-2andSNC-4) have even worse graph processingperformance, since benchmarks in graphBIG do not haveNUMA-awareness and have to pay huge penalty for frequentcommunications between different NUMA regions. SNC-4offers four NUMA regions, therefore, compared to SNC-2,it degrades the graph application performance more by largeoverhead inter-NUMA-region communications.

BFSSKCGCTCPR

0 20 40 60 80 100execution time breakdown (%)

Front-End Bound Bad Speculation Retiring Back-End Bound

Fig. 13. KNL execution time breakdown (with twitter7).

BF SS KC GC TC PR0.4

0.6

0.8

1.0no

rmal

ized

perf.

100K 1M 10M

(a) Vertex (normalized to 100K).

BF SS KC GC TC PR0.9111

1.21.41.61.8222

norm

alize

d pe

rf. 10 20 30

(b) Degree (normalized to 10).

BF SS KC GC TC PR0.40.60.81.01.21.4

norm

alize

d pe

rf. erdos-renyi real-world highly-skewed

(c) Skewness (normalized to Erdos-Rényi).Fig. 14. The KNL performance with various R-MAT datasets.

G. Execution Time Breakdown

To understand bottlenecks of applications, we show KNLexecution time breakdown of all benchmarks with twitter7graph input in Figure 13.Bad Speculationis the time stall dueto branch mis-prediction.Retiring denotes the time occupiedby the execution of useful instructions.Front-End Boundindicates the time spent by fetching and decoding instructions,while Back-End Boundmeans the waiting time due to a lackof required resources for accepting more instructions in theback-end of the pipeline, e.g., data cache misses and mainmemory accesses. It is well known that graph applications areextremely memory intensive and have irregular data accesses.The breakdown of execution time on KNL also supports thisobservation. For KC and GC,> 90% execution time is usedto wait for back-end stalls. Although in Figure 10, the 16GBMCDRAM-based cache improves the performance of KC andGC by > 10% on average, we anticipate a larger capacityMCDRAM-based cache can further boost their performance,especially when processing large graph datasets.

H. Dataset Sensitivity Analysis

We generated scale-free graphs (R-MAT [6]) with varyingnumbers of vertices, average vertex degrees and vertex dis-tributions by PaRMAT generator [14]. The dataset sensitivitystudies on KNL are shown in Figure 14. Among three R-MAT parameters (a, b and c) [6], we always enforcedb = c

for symmetry. To produce different skewnesses, we set theratio (skewness) betweena and b to 1 (Erdos-Rényi model),3 (real-world) and 8 (highly-skewed). In Figure 14(a), wefixed the average vertex degree to 20 and theskewnessto 3.With an increasing number of vertices, the performance of allbenchmarks degrades more or less, mainly because DTLB andL2 miss rates are increased by larger graph sizes. We fixed thevertex number to 1M and theskewnessto 3 in Figure 14(b). Allapplications demonstrate better performance with enlargingaverage vertex degree, since more edges per vertex lead toincreased L2 hit rate. Particularly, the performance of TCand PR increases more obviously, since they operate onneighbor sets in vertex properties and higher vertex degreebrings more accesses within vertices. In Figure 14(c), we setthe number of vertices and the average vertex degree to 1Kand 20 respectively. And we explored theskewnessamong1, 3 and 8. For traversal-based applications, BF and SS, asgraph skewness increases and graph eccentricity decreases,the application performance increases. Higher-skewed graphshave smaller diameter resulting in faster traversals. On thecontrary, the other graph benchmarks suffer from severer load

imbalance and throughput degradation, when their datasetsaremore skewed.

VI. CONCLUSION AND FUTURE WORK

In this paper, we present a performance and power charac-terization study to show the potential of KNL MIC processorson parallel graph processing. We further study the impactof KNL architectural innovations, such as many OoO cores,VPUs, cache clustering modes and MCDRAM, on the perfor-mance of multi-threaded graph applications. To fully utilizeKNLs, in future, we need to overcome challenges from bothhardware angle and software perspective.

First, from hardware angle, KNL supplies many architec-tural features that can be configured by knobs. Different graphapplications may favor different configurations. For instance,different graph benchmarks require distinctive numbers ofthreads to achieve the best performance. Furthermore, someapplications benefit from high bandwidth MCDRAM, othersmay be improved by low latency DDR4 DRAM. Therefore,it is vital to have auto-tuning tools to search the optimalconfiguration of these knobs to achieve the best performanceon KNLs. Previous works propose exhaustive iteration-basedoptimizations [9], [16] and machine-learning-based tuningtechniques [11], [18], [33]. For KNLs, we believe that fu-ture auto-tuning schemes have to consider not just the MICarchitecture, but the heterogeneous main memory system.

Second, from software perspective, though a state-of-the-artmulti-threaded graph framework fully optimized for traditionalmulti-core CPUs can run on KNLs, we observe that hard-ware resources such as VPUs are underutilized and advancedsoftware-managed architectural features, e.g., the SNC-4cacheclustering mode, may even hurt the performance of graphapplications. Previous efforts [7], [13] optimize graph datastructures and primitives to better utilizeAVX512instructionsand vectorize graph applications like breadth first search [13],page rank [8] and graph coloring [10]. In future, instead ofoptimizing a single benchmark, we need to create a fully vec-torized graph framework offeringAVX512friendly primitivesto support a wide variety of graph applications on KNLs.Moreover, we should incorporate a OS-based [21] or library-based [20] NUMA-aware memory management technique intofuture graph frameworks, so that the graph applications canbenefit from the lowest local memory access latency providedby SNC-4 without incurring large communication overheadbetween NUMA regions. The future graph frameworks onKNLs also need to be rewritten with heterogeneous memorysupporting libraries such as MEMKIND to allocate latency

sensitive pages to DDR4 DRAMs and bandwidth sensitivepages to MCDRAMs.

ACKNOWLEDGMENT

We gratefully acknowledge support from the Intel ParallelComputing Center (IPCC) grant, NSF OCI-114932 and CIF-DIBBS 143054. We appreciate the support from IU PHI,FutureSystems team and ISE Modelling and Simulation Lab.

REFERENCES

[1] M. Ahmad, F. Hijaz, Q. Shi, and O. Khan, “Crono: A benchmark suitefor multithreaded graph algorithms executing on futuristic multicores,”in IEEE International Symposium on Workload Characterization, 2015,pp. 44–55.

[2] M. Ahmad and O. Khan, “Gpu concurrency choices in graph analytics,”in IEEE International Symposium on Workload Characterization, 2016,pp. 1–10.

[3] T. Barnes, B. Cook, J. Deslippe, D. Doerfler, B. Friesen, Y. He, T. Kurth,T. Koskela, M. Lobet, T. Malas, L. Oliker, A. Ovsyannikov, A.Sarje,J. L. Vay, H. Vincenti, S. Williams, P. Carrier, N. Wichmann, M.Wagner,P. Kent, C. Kerr, and J. Dennis, “Evaluating and optimizing the nerscworkload on knights landing,” inInternational Workshop on Perfor-mance Modeling, Benchmarking and Simulation of High PerformanceComputer Systems, 2016, pp. 43–53.

[4] S. Beamer, K. Asanovic, and D. Patterson, “Locality exists in graphprocessing: Workload characterization on an ivy bridge server,” in IEEEInternational Symposium on Workload Characterization, 2015, pp. 56–65.

[5] M. Burtscher, R. Nasre, and K. Pingali, “A quantitative study of irregularprograms on gpus,” inIEEE International Symposium on WorkloadCharacterization, 2012, pp. 141–151.

[6] D. Chakrabarti, Y. Zhan, and C. Faloutsos, “R-MAT: A recursive modelfor graph mining,” inSIAM International Conference on Data Mining,2004.

[7] L. Chen, X. Huo, B. Ren, S. Jain, and G. Agrawal, “Efficientand simpli-fied parallel graph processing over cpu and mic,” inIEEE InternationalParallel and Distributed Processing Symposium, 2015, pp. 819–828.

[8] L. Chen, P. Jiang, and G. Agrawal, “Exploiting recent simdarchitecturaladvances for irregular applications,” inInternational Symposium onCode Generation and Optimization, 2016, pp. 47–58.

[9] Y. Chen, S. Fang, Y. Huang, L. Eeckhout, G. Fursin, O. Temam,andC. Wu, “Deconstructing iterative optimization,”ACM Transactions onArchitecture and Code Optimization, vol. 9, no. 3, pp. 21:1–21:30, Oct.2012.

[10] M. Deveci, E. G. Boman, K. D. Devine, and S. Rajamanickam, “Par-allel graph coloring for manycore architectures,” inIEEE InternationalParallel and Distributed Processing Symposium, 2016, pp. 892–901.

[11] A. Ganapathi, K. Datta, A. Fox, and D. Patterson, “A casefor machinelearning to optimize multicore performance,” inUSENIX Conference onHot Topics in Parallelism, 2009, pp. 1–1.

[12] J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin, “Pow-ergraph: Distributed graph-parallel computation on natural graphs,” inUSENIX Conference on Operating Systems Design and Implementation,2012, pp. 17–30.

[13] P. Jiang, L. Chen, and G. Agrawal, “Reusing data reorganizationfor efficient simd parallelization of adaptive irregular applications,” inInternational Conference on Supercomputing, 2016, pp. 16:1–16:10.

[14] F. Khorasani, R. Gupta, and L. N. Bhuyan, “Scalable simd-efficientgraph processing on gpus,” inInternational Conference on ParallelArchitectures and Compilation Techniques, 2015, pp. 39–50.

[15] F. Khorasani, K. Vora, R. Gupta, and L. N. Bhuyan, “Cusha: Vertex-centric graph processing on gpus,” inInternational Symposium on High-performance Parallel and Distributed Computing, 2014, pp. 239–252.

[16] P. A. Kulkarni, D. B. Whalley, G. S. Tyson, and J. W. Davidson, “Prac-tical exhaustive optimization phase order exploration and evaluation,”ACM Transactions on Architecture and Code Optimization, vol. 6, no. 1,pp. 1:1–1:36, Apr. 2009.

[17] A. Kyrola, G. Blelloch, and C. Guestrin, “Graphchi: Large-scale graphcomputation on just a pc,” inUSENIX Conference on Operating SystemsDesign and Implementation, 2012, pp. 31–46.

[18] H. Leather, E. Bonilla, and M. O’Boyle, “Automatic feature generationfor machine learning based optimizing compilation,” inIEEE/ACMInternational Symposium on Code Generation and Optimization, 2009,pp. 81–91.

[19] J. Leskovec and A. Krevl, “SNAP Datasets: Stanford large networkdataset collection,” http://snap.stanford.edu/data, Jun. 2014.

[20] S. Li, T. Hoefler, and M. Snir, “Numa-aware shared-memory collec-tive communication for mpi,” inInternational Symposium on High-performance Parallel and Distributed Computing, 2013, pp. 85–96.

[21] T. Li, D. Baumberger, D. A. Koufaty, and S. Hahn, “Efficient operat-ing system scheduling for performance-asymmetric multi-core architec-tures,” in ACM/IEEE Conference on Supercomputing, 2007, pp. 53:1–53:11.

[22] Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola,and J. M.Hellerstein, “Distributed graphlab: A framework for machinelearningand data mining in the cloud,”Proceedings of the VLDB Endowment,vol. 5, no. 8, pp. 716–727, APR 2012.

[23] G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser,and G. Czajkowski, “Pregel: A system for large-scale graph processing,”in ACM SIGMOD International Conference on Management of Data,2010, pp. 135–146.

[24] J. D. McCalpin, “Memory latency on the intel xeon phi x200knl proces-sor,” http://sites.utexas.edu/jdm4372/tag/memory-latency/, Dec. 2016.

[25] R. C. Murphy, K. B. Wheeler, B. W. Barrett, and J. A. Ang, “Introducingthe graph 500,” inCray User’s Group (CUG), 2010.

[26] L. Nai, Y. Xia, I. G. Tanase, H. Kim, and C.-Y. Lin, “Graphbig:Understanding graph computing in the context of industrial solutions,” inInternational Conference for High Performance Computing,Networking,Storage and Analysis, 2015, pp. 69:1–69:12.

[27] D. Nguyen, A. Lenharth, and K. Pingali, “A lightweight infrastructurefor graph analytics,” inACM Symposium on Operating Systems Princi-ples, 2013, pp. 456–471.

[28] M. A. O’Neil and M. Burtscher, “Microarchitectural performance char-acterization of irregular gpu kernels,” inIEEE International Symposiumon Workload Characterization, 2014, pp. 130–139.

[29] A. Sodani, “Knights landing (knl): 2nd generation intel R© xeon phiprocessor,” inIEEE Hot Chips Symposium, 2015, pp. 1–24.

[30] T. Suzumura, K. Ueno, H. Sato, K. Fujisawa, and S. Matsuoka, “Per-formance characteristics of graph500 on large-scale distributed environ-ment,” in IEEE International Symposium on Workload Characterization,2011, pp. 149–158.

[31] I. Tanase, Y. Xia, L. Nai, Y. Liu, W. Tan, J. Crawford, andC.-Y. Lin, “Ahighly efficient runtime and graph library for large scale graph analytics,”in Workshop on GRAph Data Management Experiences and Systems,2014.

[32] Y. Tian, A. Balmin, S. A. Corsten, S. Tatikonda, and J. McPherson,“From "think like a vertex" to "think like a graph",”Proceedings of theVLDB Endowment, vol. 7, no. 3, pp. 193–204, Nov. 2013.

[33] A. Tiwari, C. Chen, J. Chame, M. Hall, and J. K. Hollingsworth, “Ascalable auto-tuning framework for compiler optimization,” in IEEEInternational Symposium on Parallel&Distributed Processing, 2009, pp.1–12.

[34] J. Treibig, G. Hager, and G. Wellein, “Likwid: A lightweightperformance-oriented tool suite for x86 multicore environments,” inInternational Workshop on Parallel Software Tools and ToolInfrastruc-tures, 2010.

[35] Y. Wang, A. Davidson, Y. Pan, Y. Wu, A. Riffel, and J. D. Owens,“Gunrock: A high-performance graph processing library on the gpu,”in ACM SIGPLAN Symposium on Principles and Practice of ParallelProgramming, 2015, pp. 265–266.

[36] Q. Xu, H. Jeon, and M. Annavaram, “Graph processing on gpus: Whereare the bottlenecks?” inIEEE International Symposium on WorkloadCharacterization, 2014, pp. 140–149.

[37] J. Zhong and B. He, “Medusa: A parallel graph processingsystem ongraphics processors,”ACM SIGMOD Record, vol. 43, no. 2, pp. 35–40,Dec. 2014.