1.Linked Data Query ProcessingTutorial at the 22nd International World Wide Web Conference (WWW 2013)May 14, 2013http://db.uwaterloo.ca/LDQTut2013/4. Execution ProcessOlaf HartigUniversity of Waterloo

2. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 2Outline Separated Execution Integrated Execution General Idea and Properties Push-Based Implementation [LT10, LT11] Link Traversal Based Query Execution 3. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 3 clearly separatedata retrievalandresult constructioninto twoconsecutive phasesSeparated Execution ApproachesQuery-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie244912 4. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 4Properties of Separated ExecutionQuery-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie244912 Advantage: straightforward to implement Can be combined with anysource selection strategy A traditional query executionplan might then be used forconstructing the result Downside: First solutions cannotbe reported before data retrievalhas been completed 5. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 5Query-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie2449Integrated Execution Approaches intertwinedata retrieval andresult construction 6. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 6Query-local datahttp://mdb.../Paul http://geo.../Berlinhttp://mdb.../Ric http://geo.../Rome?loc?actorGET http://.../movie2449Integrated Execution Approaches Implementations may reportfirst solutions early For monotonic queries Implementations may processdata in a streaming manner May require less query-localmemory (b/c query-local datasetneed not be materialized) Can also be combined with anysource selection strategy intertwinedata retrieval andresult construction 7. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 7Outline Separated Execution Integrated Execution General Idea and Properties Push-Based Implementation [LT10, LT11] Link Traversal Based Query Execution 8. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 8Push-Based ImplementationTP Operator TP OperatorTP OperatorJoinOperatorJoinOperator Each operator runs as anasynchronous thread Connected viamessage queuesData RetrievalOperator 9. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 9Push-Based ImplementationTP Operator TP OperatorTP OperatorJoinOperatorJoinOperator Each operator runs as anasynchronous thread Connected viamessage queues Data retrieval operator May apply any possiblesource selection strategy Usually, implemented viamultiple lookup threads Pushes any incomingmatching triple to thecorresponding triplepattern operatorData RetrievalOperator 10. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 10Push-Based Implementation (Example)TP Operator( ?p, interested in, ?i )TP Operator( ?p, affiliated with, orgax )TP Operator( ?i, rdf:type, Book )JoinOperatorJoinOperatorData RetrievalOperator 11. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 11Push-Based Implementation (Example)TP Operator( ?p, interested in, ?i )TP Operator( ?p, affiliated with, orgax )TP Operator( ?i, rdf:type, Book )JoinOperatorJoinOperatorData RetrievalOperator{ ?p alice }( alice, affiliated with, orgax )( alice, affiliated with, orgax ) 12. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 12Push-Based Implementation (Example)TP Operator( ?p, interested in, ?i )TP Operator( ?p, affiliated with, orgax )TP Operator( ?i, rdf:type, Book )JoinOperatorJoinOperatorData RetrievalOperator{ ?p alice }( alice, interested in, yoga ){ ?p alice, ?i yoga }( alice, interested in, yoga ){ ?p alice, ?i yoga } 13. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 13Symmetric Hash Join (SHJ)SHJ Operator# Bucket1 {?p alice,?i yoga}23 {?p bob,?i tea}, {?p bob,?i chess}# Bucket1 {?p alice,?o orgax}2 {?p eve,?o orgax}3 Maintains a hash table for each input Same hash function for both tables Each bucket contains a set of input solutions 14. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 14How SHJ Processes Incoming SolutionSHJ Operator# Bucket1 {?p alice,?i yoga}23 {?p bob,?i tea}, {?p bob,?i chess}# Bucket1 {?p alice,?o orgax}2 {?p eve,?o orgax}3 Hash the value that input binds to the join variableinput = {?p bob,?o acme} 15. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 15How SHJ Processes Incoming SolutionSHJ Operator# Bucket1 {?p alice,?i yoga}23 {?p bob,?i tea}, {?p bob,?i chess}# Bucket1 {?p alice,?o orgax}2 {?p eve,?o orgax}3 Hash the value that input binds to the join variable Insert input into the corresponding input hash tablehfct(bob) = 3input = {?p bob,?o acme} 16. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 16How SHJ Processes Incoming SolutionSHJ Operator# Bucket1 {?p alice,?i yoga}23 {?p bob,?i tea}, {?p bob,?i chess}# Bucket1 {?p alice,?o orgax}2 {?p eve,?o orgax}3 {?p bob,?o acme} Hash the value that input binds to the join variable Insert input into the corresponding input hash table Probe into the other hash table to find join candidates Merge input with each join candidate, push results to output{?p bob,?o acme,?i tea}hfct(bob) = 3{?p bob,?o acme,?i chess}input = {?p bob,?o acme} 17. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 17Outline Separated Execution Integrated Execution General Idea and Properties Push-Based Implementation [LT10, LT11] Link Traversal Based Query Execution 18. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 18Link Traversal Based Query Execution is the combination ofintegrated execution andlive exploration(i.e., the push-based approach can beused as an implementation of LTBQE)An even closer combination presents thefollowing pull-based implementation approach ... 19. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 19?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type QuerySeed: Pipeline of Link Traversing IteratorsI3I2I1 20. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 20?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type Querytp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3Seed: Pipeline of Link Traversing Iterators 21. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 21?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type QuerySeed: tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3query-localdatasetPipeline of Link Traversing Iterators 22. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 22?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type QuerySeed: tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3query-localdatasetPipeline of Link Traversing Iterators 23. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 23?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type QuerySeed: tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3query-localdatasetPipeline of Link Traversing Iterators 24. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 24query-localdatasetNext?tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3Pipeline of Link Traversing Iterators 25. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 25query-localdatasetNext?Next?tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3Pipeline of Link Traversing Iterators 26. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 26query-localdatasetNext?Next?Next?tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3Pipeline of Link Traversing Iterators 27. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 27Next?Next?Next?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3tp1= ( ?p , ex:affiliated_with , ) I1: ex:affiliated_with :query-localdatasetPipeline of Link Traversing Iterators 28. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 28Next?tp1= ( ?p , ex:affiliated_with , ) I1: ex:affiliated_with :query-localdataset{ ?p = }Next?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3Pipeline of Link Traversing Iterators 29. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 29tp1= ( ?p , ex:affiliated_with , ) I1query-localdataset{ ?p = }Next?Next?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3Pipeline of Link Traversing Iterators 30. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 30Next?Next?tp3= ( ?b , rdf:type , ) I3tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2query-localdataset{ ?p = }Pipeline of Link Traversing Iterators 31. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 31Next?Next?tp3= ( ?b , rdf:type , ) I3tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2query-localdataset{ ?p = }: ex:interested_in :Pipeline of Link Traversing Iterators 32. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 32Next?Next?tp3= ( ?b , rdf:type , ) I3tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = }: ex:interested_in :query-localdatasetPipeline of Link Traversing Iterators 33. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 33Next?tp3= ( ?b , rdf:type , ) I3tp1= ( ?p , ex:affiliated_with , ) I1tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = }: ex:interested_in :query-localdataset{ ?p = , ?b = }Pipeline of Link Traversing Iterators 34. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 34tp1= ( ?p , ex:affiliated_with , ) I1Next?{ ?p = }query-localdatasettp3= ( ?b , rdf:type , ) I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = , ?b = }Pipeline of Link Traversing Iterators 35. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 35tp1= ( ?p , ex:affiliated_with , ) I1Next?{ ?p = }query-localdatasettp3= ( ?b , rdf:type , )tp3 = ( , rdf:type , )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = , ?b = }Pipeline of Link Traversing Iterators 36. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 36tp1= ( ?p , ex:affiliated_with , ) I1Next?{ ?p = }tp3= ( ?b , rdf:type , )tp3 = ( , rdf:type , )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = , ?b = }query-localdataset: rdfs:subClassOf :Pipeline of Link Traversing Iterators 37. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 37tp1= ( ?p , ex:affiliated_with , ) I1Next?{ ?p = }tp3= ( ?b , rdf:type , )tp3 = ( , rdf:type , )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = , ?b = }query-localdataset: rdf:type :Pipeline of Link Traversing Iterators 38. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 38tp1= ( ?p , ex:affiliated_with , ) I1Next?{ ?p = }tp3= ( ?b , rdf:type , )tp3 = ( , rdf:type , )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = , ?b = }: rdf:type :query-localdatasetPipeline of Link Traversing Iterators 39. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 39{ ?p = , ?b = }tp1= ( ?p , ex:affiliated_with , ) I1{ ?p = }tp3= ( ?b , rdf:type , )tp3 = ( , rdf:type , )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2{ ?p = , ?b = }query-localdatasetPipeline of Link Traversing Iterators 40. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 40tp1= ( ?p , ex:affiliated_with , ) I1tp3= ( ?b , rdf:type , )tp3 = ( , rdf:type , )I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2Pipeline of Link Traversing Iterators{ ?p = }{ ?p = , ?b = }Next?query-localdataset 41. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 41tp1= ( ?p , ex:affiliated_with , ) I1tp3= ( ?b , rdf:type , ) I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2Pipeline of Link Traversing Iterators{ ?p = }Next?Next?query-localdataset 42. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 42tp1= ( ?p , ex:affiliated_with , ) I1tp3= ( ?b , rdf:type , ) I3tp2= ( ?p , ex:interested_in , ?b )tp2 = ( , ex:interested_in , ?b )I2Pipeline of Link Traversing Iterators{ ?p = }Next?Next?query-localdataset 43. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 43 Deeply intertwines integrated execution + live exploration Data retrieval is a side-effect of calling link traversing iterators No need for a separate data retrieval operator Makes adding LTBQE to existing SPARQL engines easy Because those usually use iterators for execution Caveat: does not achieve the full flexibilitythat is possible with LTBQE Iterators evaluate the triple patterns in a fixed order Iterators discard each input solution after using itPipeline of Link Traversing Iterators 44. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 44?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type QuerySeed: tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?b , rdf:type , ) I3tp1= ( ?p , ex:affiliated_with , ) I1Alternative Execution Order 45. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 45?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type QuerySeed: tp2= ( ?p , ex:interested_in , ?b ) I2Alternative Execution Ordertp3= ( ?p , ex:affiliated_with , ) I3tp1= ( ?b , rdf:type , ) I1 46. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 46?p ex:affiliated_with ?p ex:interested_in ?b?b rdf:type QuerySeed: tp2= ( ?p , ex:interested_in , ?b ) I2query-localdatasetIterator Based Executiontp1= ( ?b , rdf:type , ) I1tp3= ( ?p , ex:affiliated_with , ) I3 47. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 47tp3= ( ?p , ex:affiliated_with , ) I3tp2= ( ?p , ex:interested_in , ?b ) I2tp1= ( ?b , rdf:type , ) I1: ex:affiliated_with :query-localdatasetAlternative Execution Order 48. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 48Next?: ex:affiliated_with :query-localdatasetNext?tp1= ( ?b , rdf:type , ) I1Alternative Execution OrderNext?tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?p , ex:affiliated_with , ) I3 49. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 49Next?: ex:affiliated_with :query-localdatasetNext?tp1= ( ?b , rdf:type , ) I1Alternative Execution OrderEND!tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?p , ex:affiliated_with , ) I3 50. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 50query-localdatasettp1= ( ?b , rdf:type , ) I1tp2= ( ?p , ex:interested_in , ?b ) I2tp3= ( ?p , ex:affiliated_with , ) I3END!END!END!Computed queryresult may dependon the order of triple patterns= logical query execution planAlternative Execution Order 51. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 51Outline Separated Execution Integrated Execution General Idea and Properties Push-Based Implementation [LT10, LT11] Link Traversal Based Query ExecutionNext part: 5. Query Planning and Optimization ... 52. WWW 2013 Tutorial on Linked Data Query Processing [ Execution Process ] 52These slides have been created byOlaf Hartigfor theWWW 2013 tutorial onLink Data Query ProcessingTutorial Website: http://db.uwaterloo.ca/LDQTut2013/This work is licensed under aCreative Commons Attribution-Share Alike 3.0 License(http://creativecommons.org/licenses/by-sa/3.0/)