an i online graph partitioning algorithm for distributed graph … · daniel needs distributed...
TRANSCRIPT
![Page 1: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/1.jpg)
an Incremental Online Graph Partitioning algorithm for
distributed graph databases
Dong Dai*, Wei Zhang, Yong Chen
IOGP
![Page 2: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/2.jpg)
Workflow of The Presentation
6/30/17 HPDC 2017, Washington D.C. 2
A Use Case
OLTP vs. OLAP
RethinkGraph
Partitioning
Modeling and
Analysis
IOGP Design Idea
IOGP DetailsEvaluation
Setup
Partition Quality
OLTPPerformance
![Page 3: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/3.jpg)
A Use Case
• MeetDanielfrom“All-About-HPC”LLC• HewantstobuildaprovenancesystemforHPC
• Hetracesallprovenance-relevantevents• Hemodelsandstoresthemintoaprovenancegraph• Heoffersusersgraphinterfacestoqueryprovenance
6/30/17 HPDC 2017, Washington D.C. 3
New events inserted continuously
Build large graphs with properties
Thousands of clients query it
Typical OLTP Workloads
![Page 4: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/4.jpg)
OLTP vs. OLAP
• Wehavetwosetsoftoolstohandlegraphs• GraphDatabases• short,finishinmilliseconds• touchasmallpartofthegraph• measuredbytimecostofeachtransaction
• GraphProcessingEngines• longer,finishinseconds,minutes• touchalargepartoforthewholegraph• measuredbysystemthroughput
6/30/17 HPDC 2017, Washington D.C. 4
OLTP
OLAP
Daniel needs distributed graph databases for his OLTP opsBut, how he partitions the graphs?
![Page 5: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/5.jpg)
Rethink the Graph Partitioning
•Manygraphpartitioningalgorithms•Multi-levelscheme• METIS,Chaco,PMRSB,Scotch,ParMetis,Pt-Scotch,etc.
• Heuristicsingle-passmethods• LDG,Fennel,2D-Grid,Vertex-Cut,etc.
• Onlinepartitioningalgorithms• Hashing,Leopard,etc.
6/30/17 HPDC 2017, Washington D.C. 5
Not Designed for OLTP Workloads
![Page 6: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/6.jpg)
Rethink the Graph Partitioning
• WhatdoesOLTPneed?
• Responsequicklywheninsertionhappens• Itneedstofinishfast(inms)• Notimetowaitformulti-levelschemesorevenre-partitioning
• Responsewhileknownothingaboutthevertex• Theinsertionmaybe“random”• Donotassumeknowconnectivityofinsertedvertex
• Measurementofagoodpartitioning• OLTPcaresresponsetimeofeachoperation• Onlyminimizingedge-cuts/maximizingthebalanceisnotenough• Thinkaboutagraphwithnequal-size subgraphs
6/30/17 HPDC 2017, Washington D.C. 6
![Page 7: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/7.jpg)
Modeling and Analysis
• Genericdistributedgraphdatabasemodel• directed/undirectedgraphs• bi-directionaccessesonthegraphs• queryablepropertiesonverticesandedges
6/30/17 HPDC 2017, Washington D.C. 7
![Page 8: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/8.jpg)
Factors of OLTP Performance
• Single-PointAccessPerformance• One-hop:ifclientsknowthelocationofquerieddata• Itsavestimeforqueryinglocationinformation
• TraversalPerformance(Edge-CutandVertex-Cut)
6/30/17 HPDC 2017, Washington D.C. 8
![Page 9: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/9.jpg)
IOGP Core Idea
• Leveragedeterministichashingtoplacenewvertices• fastandeasytocalculate• enableone-hopaccess• needno infotoconductpartitioning
• Incrementallyreassignverticestobetterlocation• continuouslygetbetterlocalitywithincreasingknowledge
• Forvertexistoolarge,changefromedge-cuttovertex-cut• increasetheparallelism• improveOLTPresponsetime
6/30/17 HPDC 2017, Washington D.C. 9
![Page 10: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/10.jpg)
IOGP - Quiet Stage
• QuietStageisthedefaultstage• avertexv isinserted,itwillbeplacedbasedonHashing• anedgeu->v isinserted,itwillbeplacedtogetherwithitsconnectedvertices
6/30/17 HPDC 2017, Washington D.C. 10
v à u
v
![Page 11: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/11.jpg)
Vertex Reassign Stage
• Whenweknewenoughconnectivityofavertexv• moveittothepartitionthatstoresthemostofitsneighbors• Aheuristicfunction(derivedfromFennel)
6/30/17 HPDC 2017, Washington D.C. 11
v à u
v|u1, u2, u3, ...
![Page 12: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/12.jpg)
Edge Splitting Stage
• Whenavertexbecomesextremelylarge• Edge-cutleadstosignificantperformancedegradation• Splitalledgestoincreasetheparallelismofdataaccesses
6/30/17 HPDC 2017, Washington D.C. 12
![Page 13: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/13.jpg)
IOGP Details
• We need per-vertex metadata• To record vertex location• To know vertex degree or it has been split• To efficiently calculate
• Counters
6/30/17 HPDC 2017, Washington D.C. 13
split(v) ...,...loc(v)
alo(v)
ali(v)
plo(v)
pli(v)
plo(v)
pli(v)
split(v)loc(v)
counters(v)
Original Location
![Page 14: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/14.jpg)
Update Counters after Reassignment
1. Updateloc(v) intheoriginalserver
2. InserverSu (vismovingoutofit)• v’sactuallocalityturnsintopotentiallocality• v’slocalneighbors
• reduceactuallocalityby1becausevisnotlocalanymore
3. InserverSk (vismovingintoit)• v’spotentiallocalityturnsintoactuallocality• v’slocalneighbors
• increasetheiractuallocalityby1becausev islocaltothemnow
6/30/17 HPDC 2017, Washington D.C. 14
![Page 15: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/15.jpg)
Optimization: Timing of Vertex Reassignment
• Vertexreassignmentistime-consuming• Observation• moreedges,morestableconnectivity;• lessreassignmentisneeded
• Design• deferringvertexreassignmentuntilitsconnectivitystabilizes• reducingvertexreassignmentfrequencywithmoreedges
• Implementation• Startreassignmentuntilitsdegreeishigherthank• Checkforreassignmentwhenitsdegreereaches[2*k,4*k,...,2l*k...]
6/30/17 HPDC 2017, Washington D.C. 15
![Page 16: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/16.jpg)
Optimization: Asynchronous Data Movement
• Vertexreassignmentandedgesplitting• syncdatamovementmaystaleOLTPoperations• optimization:asynchronousdatamovement
• Edgesplittingexample• Updatein-memorycounters• addvertexinpendingmovementqueue• Serverstartstorejectnewedges
• Datamayindifferentlocationsduringmovement• originalserver,newserver,andmaybeboth• clientsneedtoreadmultiplecopiesandaggregatethem• Prefertheonefrom“newserver”
6/30/17 HPDC 2017, Washington D.C. 16
![Page 17: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/17.jpg)
Evaluation Setup
• AllevaluationswereconductedontheCloudLab cluster• 32serversforback-endstorage
• WechosevariousdatasetsfromSNAPforevaluations• scalesfrom200Kedgestoover100Medges• fromvariousdomains,formingdifferentshapes
• WeevaluatedIOGPonadistributedgraphdatabaseprototype,namelySimpleGDB(https://github.com/daidong/simplegdb-Java)• Aprototypesystemhasbeenusedinseveralresearchprojects• Forfaircomparisonandcontrollableoptimizations
6/30/17 HPDC 2017, Washington D.C. 17
![Page 18: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/18.jpg)
Partitioning Quality
METIS is not always balancedOthers (including IOGP) are well balanced
6/30/17 HPDC 2017, Washington D.C. 18
• METIS runs directly on the final graph• Fennel runs in random order
METIS is still the bestIOGP achieves very well
![Page 19: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/19.jpg)
Continuous Refinement of IOGP
• Usingsimilarheuristic,IOGPperformsbetterthanFennelduetocontinuousrefinement
6/30/17 HPDC 2017, Washington D.C. 19
![Page 20: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/20.jpg)
Key Tunable Parameters
Reassignment Threshold
somewhere around average degree of the graph
6/30/17 HPDC 2017, Washington D.C. 20
Split Threshold
Relevant with 1) hardware; 2) scale of the database cluster; 3) vertex degree and property size
![Page 21: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/21.jpg)
Memory Footprint
• HowmanymemoryisusedinIOGPtokeepthein-memorycounters?
6/30/17 HPDC 2017, Washington D.C. 21
![Page 22: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/22.jpg)
OLTP Performance
Graph Insertionless than 10% overhead
6/30/17 HPDC 2017, Washington D.C. 22
Graph TravelAs much as 2x benefits
![Page 23: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/23.jpg)
Conclusion & Future Work
• OLTPworkloadsofdistributedgraphdatabasesrequirenewsetofgraphpartitioningalgorithms.
• IOGPachieveslessthan10%overheadsoninsertion,andasmuchas2xperformanceimprovementonqueries.
• Therearestilllotsofthingscanbedone• Reducetheoverheadsofvertexreassignments• Partitioningthegraphsbasedontheexactworkloadpatterns• ...
6/30/17 HPDC 2017, Washington D.C. 23
![Page 24: an I Online Graph Partitioning algorithm for distributed graph … · Daniel needs distributed graph databases for his OLTP ops But, how he partitions the graphs? Rethink the Graph](https://reader033.vdocuments.us/reader033/viewer/2022053016/5f19c3857550b9351058a152/html5/thumbnails/24.jpg)
Thanks and Questions
6/30/17 HPDC 2017, Washington D.C. 24