using mongodb as a high performance graph database

Thursday, 21 June 12

Using MongoDB as a high performance graph database

MongoDB UK, 20th June 2012

Chris Clarke CTO, Talis Education Limited

Who is talis?

Using mongo about 8 months (since 2.0)5 months in production

What this talk not about

A blueprint for what you should doA pitch to encourage you to take our approachProviding or proving performance benchmarks Evangelism for the semantic web or linked dataEncouraging you to contribute/download/use an open source projectOptimised for your use case

Although we can talk to you about any of the above (see me after)

So, what is this talk about?

Our journey of using MongoDB as a high performance graph databaseSpecifically the software wrapper we implemented on top of Mongo to give us a leg up in terms of scalability and performanceTo give you some ideas for how to work with graph data models if you’d like to use document databases

GRAPHS 101

ApologiesNodes and edges or Resources and propertiesReally easy to represents facts

John knows Jane

knowsJohn Jane

Ball and stick diagramsThis is an undirected graph. It implies that John knows Jane and Jane knows John. The property has no directional significance.

knowsJohn Jane

John knows JaneJane knows John

This is an undirected graph. It implies that John knows Jane and Jane knows John. The property has no directional significance.

knowsJohn Jane

John knows JaneJane ? John

This is a directed graph. The relationship is one way. To add Jane knows John we need a second property.

We will only use directed graphs from herein as they are more specific

John Janeknows

John knows JaneJane knows John

Triples + RDF 101

Subject Property Object

John knows Jane

This is a triple

Property = predicate

John knows Jane

Jane knows John

This is a second tripleThe same resource can be a subject or an object

http://xmlns.com/foaf/0.1/knows http://example.com/Janehttp://example.com/John

RDFResources and properties as URIsURIs can be dereferencedCan share common property descriptions (RDF Schemas)Here using FOAF - billions if not trillions of triples defined using FOAF

foaf:knows http://example.com/Janehttp://example.com/John

PREFIX foaf: <http://xmlns.com/foaf/0.1/>

foaf:name “John”http://example.com/John

Namespaces for readability

In RDF subjects are always urisBut objects can be literals i.e. plain textMany RDF/graph databases allow you to further type literals as dates, numbers, etc.

foaf:name “John”

http://example.com/John

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

PREFIX foaf: <http://xmlns.com/foaf/0.1/>

foaf:Personrdf:type

http://example.com/John

foaf:name “Jane”

http://example.com/Jane foaf:Personrdf:type

http://example.com/Jane

foaf:knows http://example.com/Janehttp://example.com/John

foaf:knows http://example.com/Johnhttp://example.com/Jane

Here we type John and Jane as foaf:Person using rdf:type

Note both John and Jane appear as subjects and resources

This RDF graph represents six facts

example:John

foaf:knows

example:Jane

foaf:Person

rdf:type rdf:type

“John” “Jane”

Here it is in ball and stick

FFS! I can do that in two minutes in BSON

> db.people.find(){

_id: ObjectID(‘123’),name: ‘John’knows: [ObjectID(‘456’)]

_id: ObjectID(‘456’),name: ‘Jane’knows: [ObjectID(‘123’)]

Yes, you can!Data only makes sense inside your db though

http://sheikspear.blogspot.co.uk/2011/07/simples.html

Talk over, right?We can all go home

Some useful stuff, using RDF

Lets look at some reasons why we think RDF is good

attribution

This is the linked open data cloud

Linked data is a way RDF published on the open web

Search linked data TED to hear why Tim Burness Lee cares about this

Each blob on this diagram represents an open, interlinked dataset. The lines between them represent the interlinking between data sets

Billions of public “facts” and growing exponentially from sites such as BBC, governments, Last.fm, Wikipedia

Merging data from different sources is really easy

Because the format is subject, predicate, object the shape of RDF is always the same. Because schemas are public and widely shared the same properties are used all over the place.Really easy to use this data in your own app and remix

Dataset B

foaf:Person

example:Johnexample:John

rdf:type

“John”

foaf:name

Dataset A

foaf:Person

example:John

rdf:type

“John”

foaf:name

Dataset A+B

Really easy to merge graphs“Designed in” to the data formatLots of existing tooling to do this

RDF query language: SPARQL

PREFIX foaf: <http://xmlns.com/foaf/0.1/>SELECT ?name ?emailWHERE { ?person a foaf:Person. ?person foaf:name ?name. ?person foaf:mbox ?email.}ORDER BY ?nameLIMIT 50

SPARQL is mega flexible. Lots of functions for grouping, walking graphs, pattern matching, inference, UNIONS, Geo extensions etc. etc. - all that shit. Most if not all of those datasets will have a SPARQL endpoint you can query

SELECT TabularDESCRIBE GraphASK BooleanCONSTRUCT Graph

4 main query types

PREFIX foaf: <http://xmlns.com/foaf/0.1/>SELECT ?name ?emailWHERE { ?person a foaf:Person. ?person foaf:name ?name. ?person foaf:mbox ?email.}ORDER BY ?nameLIMIT 50

FFS! That looks like SQL!

Yes it does. The WHERE clause is basically doing a shit load of joins. I’ll come back to that.

Offline conversion process

Application DB(SQL or other)

Triple store +SPARQL

Most datasets on the LOD diagram don’t exist natively as Linked data and RDF. They are post-produced.Data not held natively - so conversion script - needs to be maintained and updated every time app schema changesData not up to date (1 hour, 1 day, 1 month behind?)

Our innovation:Native Linked Data

Applications

We started working on these applications back in 2008

They are natively linked data so solve the conversion+currency issue

There is no other “format” or schema the data is stored in, it’s native RDF

When you have no schema, and you can integrate data from elsewhere on the web, it’s addictive

Our problem:FFS! For applications, we

need humongous scale and performance

Those applications becoming rather popular with our users...

sub 50ms query time

Modern web apps need speed and data scale

Out-grown triple store and SPARQL

SPARQL is very flexible and expressive. It’s also expensive SPARQL is great for data sets where the questions you can ask are limitless, but our applications need a data layer where speed is measured in single digit ms.

Complex caching (w/Memcache) to achieve performance and scalability90:10 read:write

Tripod

It’s a pod for our triplesA triple store designed for applications and scalabilityBased on Mongo

Functional requirements:• Order magnitude increase in perf/scale• Graph-orientated interface

Non-functional requirements:• Strong community

Existing code very graph orientated

Core data formatTripod API

Dealing with complex queriesTripodTables

Free text search

Walk through Tripod looking at 5 areas

{‘http://example.com/John’ : {

‘http://purl.org/dc/elements/1.1/name’ : [{

value: ‘John’,type: ‘literal’

}],‘http://purl.org/dc/elements/1.1/knows’ : [

{value: ‘http://example.com/Jane’,type: ‘uri’

},‘http://example.com/Jane’ : {

‘http://purl.org/dc/elements/1.1/name’ : [{

value: ‘Jane’,type: ‘literal’

}],‘http://purl.org/dc/elements/1.1/knows’ : [

{value: ‘http://example.com/John’,type: ‘uri’

value: ‘http://example.com/James’,type: ‘uri’

RDF/JSON - a serialisation of RDF in JSON

Neither disk space efficient or readable

full-formed properties not compatible with Mongo (dot notation)

Even single values inside an array (problems for compound indexing)

> db.CBD_people.find(){

_id: ‘http://example.com/John’,‘foaf:name’: {l: ‘John’},‘foaf:knows’: {u: ‘http://example.com/Jane’}

_id: ‘http://example.com/Jane’,‘foaf:name’: {l: ‘Jane’},‘foaf:knows’: [ {u:‘http://example.com/John’}, {u:‘http://example.com/James’}]

Same semantics

2 documents here

Concise bound descriptions - all data known about a subject, one relationship deep

One document per subject per collection, keyed (and thus enforced) by Subject URI

Property names are namespaced

CBD collections are deemed as read/write in Tripod

class MongoGraph extends SimpleGraph {

function add_tripod_array($tarray) function to_tripod_array($docId)

All of our app already uses SimpleGraph from a library called Moriarty (Google Code)

Simple extension which can ingest/output the data format on prev slide

Free text search

Walk through Tripod looking at 5 areas

interface ITripod{ public function select($query,$fields,$sortBy=null,$limit=null); public function describeResource($resource); public function describeResources(Array $resources); public function saveChanges($oldGraph, $newGraph); public function search($query);}

Almost the same as our existing data access API onto generic triple store

All of these methods return graphs, all are mega-simple queries on the CBD collections

None of these methods support joins (WHERE clause in SPARQL)

public function describeResource($resource){

$query = array(“_id”=>$resource);$bson = $this->getCollection()->findOne($query);$graph = new MongoGraph();$graph->add_tripod_data($bson);return $graph;

These methods mega simple to implement as they translate to really simple Mongo Queries on the CBD collections returning single objects

interface ITripod{ public function select($query,$fields,$sortBy=null,$limit=null); public function describeResource($resource); public function describeResources(Array $resources); public function saveChanges($oldGraph, $newGraph); public function search($query);

public function getViewForResource($resource,$viewType); public function getViewForResources(Array $resources,$viewType); public function getViews(Array $filter,$viewType);

Some extra methods to deal with complex queries involving joins

Free text search

2 things we realised when looking at our applications

DESCRIBE <http://example.com/foo> ?sectionOrItem ?resource ?document ?authorList ?author ?usedBy ?creator ?libraryNote ?publisherWHERE{

OPTIONAL {

<http://example.com/foo> resource:contains ?sectionOrItem . OPTIONAL

{ ?sectionOrItem resource:resource ?resource .

OPTIONAL { ?resource dcterms:isPartOf ?document . } OPTIONAL {

?resource bibo:authorList ?authorList . OPTIONAL { ?authorList ?p ?author . } } OPTIONAL { ?resource dcterms:publisher ?publisher . } } OPTIONAL { ?libraryNote bibo:annotates ?sectionOrItem } } . OPTIONAL { <http://example.com/foo> resource:usedBy ?usedBy } . OPTIONAL { <http://example.com/foo> sioc:has_creator ?creator }}

Typical SPARQL query in our app

9 “joins” in this query

DESCRIBE <http://example.com/foo> ?sectionOrItem ?resource ?document ?authorList ?author ?usedBy ?creator ?libraryNote ?publisherWHERE{

OPTIONAL {

<http://example.com/foo> resource:contains ?sectionOrItem . OPTIONAL

{ ?sectionOrItem resource:resource ?resource .

OPTIONAL { ?resource dcterms:isPartOf ?document . } OPTIONAL {

?resource bibo:authorList ?authorList . OPTIONAL { ?authorList ?p ?author . } } OPTIONAL { ?resource dcterms:publisher ?publisher . } } OPTIONAL { ?libraryNote bibo:annotates ?sectionOrItem } } . OPTIONAL { <http://example.com/foo> resource:usedBy ?usedBy } . OPTIONAL { <http://example.com/foo> sioc:has_creator ?creator }}

Only thing that changes at run time in this query is this URI

Flexibility of SPARQL great for developer but terrible here for system performance

Query engine needs to join 9 times! Flexibility costs us every time we run this query!

This is why we hid it behind a cache

join countfollow sequences (n times)join across databases

All the above with a condition

include certain propertiesinclude all properties

2nd thing

We only make use of minimal SPARQL

And some of these aren’t even well supported in SPARQL (sequences + join across databases)

Materialised views, generated infrequently, read often

Remember 90:10 read:update

View specifications based on a subset of SPARQL

Views are for DESCRIBE like queries where all the data is brought back in one hit (not tabular data)

{ _id: "v_resource_brief", from: "CBD_harvest", type: "http:\/\/talisaspire.com\/schema#Resource", include: ["rdf:type", "dct:subject", "dct:isVersionOf", "searchterms:usedAt", "dc:identifier"], joins: { "acorn:preferredMetadata": [], "acorn:listReferences": { include: ["acorn:list"] }, "acorn:bookmarkReferences": { include: ["acorn:bookmark"] }, "dcterms:isPartOf": [], "acorn:partReferences": { include: ["dct:hasPart"], joins: { "dct:hasPart": { joins: { "acorn:preferredMetadata": [] } } } } }}

A view specification - itself a document that can be stored in Mongo

8 keywords:

type from include joinsttl followSequence maxJoins counts

Generated by incremental MapReduce when:1) Data is changed

2) TTL expires

Tripod can take these specifications and manage views in a special collection within the DB.

They expire and are regenerated automatically (and incrementally)

Incremental map reduce inside the DB

Fast, interleaves with reads

> db.views.findOne(){ "_id" : { "rdf:resource" : "http://talisaspire.com/examples/1", "type" : "v_resource_full" }, "value" : { "graphs" : [ { "_id" : "http://talisaspire.com/examples/1", "rdf:type" : { "type" : "uri", "value" : "http://talisaspire.com/schema#Resource" } } ], "impactIndex" : [ "rdf:resource" : "http://talisaspire.com/examples/1" ] }}

This is what a view looks like

ID is a composite key of the view type and root resourceGraphs is a collection of CBDs

MongoGraph we displayed earlier can take this and represent it as a unified graph to the application

Impact index - A watch list of resources. When resources are saved the impact index is queried to find views that need invalidating

TTL is an alternative. If in viewspec timestamp is stored in view to determine when it can be invalidated

attribution

Match views to data update rate

Free text search

Tripod Tables are for larger datasets which cannot be brought back in one hit

They can be paged or have individual columns indexed for fast sort capability

SELECT ?listName ?listUri!WHERE{! ?resource bibo:isbn10 "$isbn" ! UNION! { ! ! ?resource bibo:isbn10 "$isbnLowerCase" .! }! ?item resource:resource ?resource .! UNION! {! ! ?resourcePartOf bibo:isbn10 "$isbn" .! ! UNION! ! {! ! ! ?resourcePartOf bibo:isbn10 "$isbnLowerCase" . ! ! }! ! ?resourcePartOf dct:hasPart ?resource .! ! ?item resource:resource ?resource . } ?listUri resource:contains ?item . ?listUri sioc:name ?listName . ?listUri rdf:type resource:List}LIMIT 10OFFSET 40

This is a select query that brings back a two col document

OFFSET

<?xml version="1.0"?><sparql xmlns="http://www.w3.org/2005/sparql-results#">! <head>! ! <variable name="label"/>! ! <variable name="type"/>! </head>! <results>! ! <result>! ! ! <binding name="label">! ! ! ! <literal>Tropical grassland</literal>! ! ! </binding>! ! ! <binding name="type">! ! ! ! <uri>http://purl.org/ontology/wo/TerrestrialHabitat</uri>! ! ! </binding>! ! </result>! ! <result>! ! ! <binding name="label">! ! ! ! <literal>Grassy field</literal>! ! ! </binding>! ! ! <binding name="type">! ! ! ! <uri>http://purl.org/ontology/wo/TerrestrialHabitat</uri>! ! ! </binding>! ! </result>! </results></sparql>

SPARQL SELECT results - tabular format - here in XML

> db.t_resource.findOne(){

"_id" : "http://talisaspire.com/resources/3SplCtWGPqEyXcDiyhHQpA-2","value" : {

"type" : ["http://purl.org/ontology/bibo/Book","http://talisaspire.com/schema#Resource"

],"isbn" : "9780393929690","isbn13" : [

"9780393929691","9780393929691-2",

! "9780393929691-3"],"impactIndex" : [

"http://talisaspire.com/works/4d101f63c10a6",]

This time our map reduce doesn’t create one doc as with materialised views

We get one doc per row

Free text search

Our triple store included free text search

We wanted to stream updates into Elastic Search or A N Other search solution

When documents saved, same specification language used to build Search Document Format docs and submit them to an endpoint

We like ElasticSearch but you could use Amazon CloudSearch

Limitations

Map Reduce as a non-blocking db.eval() and also to work around sync PHP programming model

PHP only for now - our web apps were PHP

To get a SPARQL endpoint we are exporting data out to Fueski - solved the mapping not the currency (for SPARQL)

Future

Node JS portUse as a server not a libraryEliminate dependancy on map reduceSpecification version controlTap into op log for stream approach into Fuseki and other locationsNamed graph supportFurther optimisation of data modelMaybe open source

That’s it

Questions?

Find us on:

Web: talisaspire.comTwitter: @talisaspireYouTube: youtube.com/user/TalisAspireFacebook: facebook.com/talisaspireSupport: support.talisaspire.com

Questions?

Find us on:

Web: talisaspire.comTwitter: @talisaspireYouTube: youtube.com/user/TalisAspireFacebook: facebook.com/talisaspireSupport: support.talisaspire.com

using mongodb as a high performance graph database

public function

public function

public function

public function

prefix foaf

limit 50thursday

sortbynull

limitnull

Documents

using mongodb and a relational database at mongodb day

mongodb - a document-based nosql database

mongodb nosql database a deep dive -mywhitepaper

getting started with mongodb - redlich · • getting...

mongodb database as storage for gpon frames

advanced database project: document stores and mongodb ·...

mongodb - an agile nosql database

neo4j + mongodb - sf graph database meetup group...

chao zhang mongodb with dlvhex. the mongodb introduction to...

what is mongodb?€¦ · the mongodb database consists of a...

json & mongodb · 2020. 10. 28. · inside a mongodb...

understanding and tuning wiredtiger, the new high...

mongodb -"alternative" database

mongodb database replication

mongodb -...

using mongodb as a graph database - 2014 redux

mongodb gridfs : database file storage

mongodb europe 2016 - graph operations with mongodb

mongo-scaling-phpnw16 - derick rethans · nosql redis...

mongodb - the database strikes back