mlw 2014 lightning talk - sparql 101
DESCRIPTION
Analyzing SPARQL Performance - a MarkLogic World 2014 Lightning Talk from John Snelson, Lead Engineer, MarkLogicTRANSCRIPT
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Analyzing SPARQL Performance John Snelson, Lead Engineer @jpcs
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 2
Find and understand the query plan Profile SPARQL execution
SPARQL AST The final execution plan
SPARQL Execution Individual index lookups executed
Analyzing Performance
SPARQL Value Frequencies Statistics used for cost analysis
SPARQL Cost Analysis Highlights of the cost optimization
Trace Flags
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 3
Setting Trace Flags
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 4
Execute SPARQL Optimization trace flags
produce logging when a query is first executed
Query plans are cached – subsequent executions of the same query don’t re-optimize
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 5
Statistics [Event:id=SPARQL Value Frequencies] sessionKey=12276218969763457904 values= () = count:3023127, sUniq:189666, pUniq:125, oUniq:504106 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> = freq:188495, s: (count:0, pUniq: 0, oUniq: 0), p: (count:188495, sUniq: 188495, oUniq: 23), o: (count:0, sUniq: 0, pUniq: 0) <http://localhost/vocabulary/bench/Inproceedings> = freq:43822, s: (count:1, pUniq: 1, oUniq: 1), p: (count:0, sUniq: 0, oUniq: 0), o: (count:43821, sUniq: 43821, pUniq: 2) <http://purl.org/dc/elements/1.1/creator> = freq:152192, s: (count:0, pUniq: 0, oUniq: 0), p: (count:152192, sUniq: 102241, oUniq: 87144), o: (count:0, sUniq: 0, pUniq: 0) <http://localhost/vocabulary/bench/booktitle> = freq:45171, s: (count:0, pUniq: 0, oUniq: 0), p: (count:45171, sUniq: 45171, oUniq: 45171), o: (count:0, sUniq: 0, pUniq: 0) <http://purl.org/dc/elements/1.1/title> = freq:104457, s: (count:0, pUniq: 0, oUniq: 0), p: (count:104457, sUniq: 104457, oUniq: 104457), o: (count:0, sUniq: 0, pUniq: 0) <http://purl.org/dc/terms/partOf> = freq:35146, s: (count:0, pUniq: 0, oUniq: 0), p: (count:35146, sUniq: 35146, oUniq: 1123), o: (count:0, sUniq: 0, pUniq: 0) <http://www.w3.org/2000/01/rdf-schema#seeAlso> = freq:67673, s: (count:0, pUniq: 0, oUniq: 0), p: (count:67673, sUniq: 67673, oUniq: 67673), o: (count:0, sUniq: 0, pUniq: 0) <http://swrc.ontoware.org/ontology#pages> = freq:95003, s: (count:0, pUniq: 0, oUniq: 0), p: (count:95003, sUniq: 95003, oUniq: 400), o: (count:0, sUniq: 0, pUniq: 0) <http://xmlns.com/foaf/0.1/homepage> = freq:102558, s: (count:0, pUniq: 0, oUniq: 0), p: (count:102558, sUniq: 102558, oUniq: 102558), o: (count:0, sUniq: 0, pUniq: 0) <http://purl.org/dc/terms/issued> = freq:47100, s: (count:0, pUniq: 0, oUniq: 0), p: (count:47100, sUniq: 47100, oUniq: 79), o: (count:0, sUniq: 0, pUniq: 0) <http://localhost/vocabulary/bench/abstract> = freq:1011, s: (count:0, pUniq: 0, oUniq: 0), p: (count:1011, sUniq: 1011, oUniq: 1011), o: (count:0, sUniq: 0, pUniq: 0)
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 6
Cost Analysis [Event:id=SPARQL Cost Analysis] sessionKey=12276218969763457904 optimize=1 r=4 t=12.5759 os=450 is=30 mutations=72 seed=2264901784365688517 initialCost=(m:1.23465e+08,io:(2.45591e+07+0),cpu:(1.25806e+06+0),mem:586608,c:53328,crd: [35146,65230.2,45171,65230.2,1123,52317,400,52317,79,1011]) [Event:id=SPARQL Cost Analysis] sessionKey=12276218969763457904 diff=-1.00471e+08 diff%=-81.3764 r=0 cost=(m:2.29936e+07,io:(1.38652e+06+3.0785e+06),cpu:(1.1499e+06+178947),mem:475658,c:53821.9,crd: [35146,53821.9,45171,35146,1123,35146,400,35146,79,1011]) [Event:id=SPARQL Cost Analysis] sessionKey=12276218969763457904 diff=-1.00516e+08 diff%=-81.4129 r=1 cost=(m:2.29486e+07,io:(1.38664e+06+3.0785e+06),cpu:(1.02167e+06+214093),mem:511608,c:53821.9,crd: [35146,53821.9,35146,35146,1123,53821.9,400,35146,79,1011]) [Event:id=SPARQL Cost Analysis] sessionKey=12276218969763457904 diff=-1.00516e+08 diff%=-81.4129 r=2 cost=(m:2.29486e+07,io:(1.38664e+06+3.0785e+06),cpu:(1.02167e+06+214093),mem:511608,c:53821.9,crd: [35146,53821.9,35146,35146,1123,35146,400,35146,79,1011]) [Event:id=SPARQL Cost Analysis] sessionKey=12276218969763457904 diff=-1.00516e+08 diff%=-81.4129 r=3 cost=(m:2.29486e+07,io:(1.38664e+06+3.0785e+06),cpu:(1.02167e+06+214093),mem:511608,c:53821.9,crd: [35146,53821.9,45171,35146,1123,35146,400,35146,79,1011]) [Event:id=SPARQL Cost Analysis] sessionKey=12276218969763457904 bestCost=(m:2.29486e+07,io:(1.38664e+06+3.0785e+06),cpu:(1.02167e+06+214093),mem:511608,c:53821.9,crd: [35146,53821.9,35146,35146,1123,53821.9,400,35146,79,1011])
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 7
Query Plan [Event:id=SPARQL AST] sessionKey=12276218969763457904 plan=SPARQLModule[ Prolog[] SPARQLSelect[SPARQLProject[order(8) GraphNode[Var inproc 0] GraphNode[Var author 1] GraphNode[Var booktitle 2] GraphNode[Var title 3] GraphNode[Var proc 4] GraphNode[Var ee 5] GraphNode[Var page 6] GraphNode[Var url 7] GraphNode[Var yr 8] GraphNode[Var abstract 9] SPARQLOrder[order(8) SORTED OrderSpec[ Variable[QName[(Unknown) yr] 8] ASCENDING EMPTY MIN] ...
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 8
Query Plan SPARQLBloomJoin[order(8,0) hash(0==0) TriplePattern[order(0,4) PSO GraphNode[Var inproc 0] GraphNode[IRI <http://purl.org/dc/terms/partOf>] GraphNode[Var proc 4]] SPARQLHashJoin[order(8,0) hash(0==0) TriplePattern[order(0,1) PSO GraphNode[Var inproc 0] GraphNode[IRI <http://purl.org/dc/elements/1.1/creator>] GraphNode[Var author 1]] SPARQLRightHashJoin[order(8,0) hash(0==0) TriplePattern[order(0,9) PSO GraphNode[Var inproc 0] GraphNode[IRI <http://localhost/vocabulary/bench/abstract>] GraphNode[Var abstract 9]] SPARQLHashJoin[order(8,0) hash(0==0) SPARQLMergeJoin[order(0,7) hash(0==0) SPARQLMergeJoin[order(0,3) hash(0==0) …
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 9
Query Plan SPARQLMergeJoin[order(0,7) hash(0==0) SPARQLMergeJoin[order(0,3) hash(0==0) TriplePattern[order(0,5) PSO GraphNode[Var inproc 0] GraphNode[IRI <http://www.w3.org/2000/01/rdf-schema#seeAlso>] GraphNode[Var ee 5]] SPARQLMergeJoin[order(0,3) hash(0==0) TriplePattern[order(0,2) PSO GraphNode[Var inproc 0] GraphNode[IRI <http://localhost/vocabulary/bench/booktitle>] GraphNode[Var booktitle 2]] SPARQLMergeJoin[order(0,3) hash(0==0) TriplePattern[order(0) OPS GraphNode[Var inproc 0] GraphNode[IRI <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>] GraphNode[IRI <http://localhost/vocabulary/bench/Inproceedings>]] SPARQLMergeJoin[order(0,3) hash(0==0) TriplePattern[order(0,6) PSO …
© COPYRIGHT 2014 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 10
Profiling
