the pathology of graph databases

TinkerPop

The Pathology of Graph Databases

Marko A. RodriguezGraph Systems Architect

http://markorodriguez.com http://twitter.com/twarko

http://tinkerpop.com

WindyCityDB - Chicago, Illinois – June 25, 2011

http://tinkerpop.com

http://windycitydb.org/

There is nothing more fascinating and utterly mind-bending than traversing a graph. Those who succumb to this data processing pattern euphorically suffer from graph pathology.

This is a case study of the Graph Addict.

Abstract

GremlinG = (V,E)

What is presented is as of Gremlin 1.1 (Released July 15, 2011)

http://gremlin.tinkerpop.com

What you are about to see may disturb you.

It has infected Neo4j, OrientDB, DEX, RDF Sail, TinkerGraph, and ReXster.

...it will infect others.

reXster

http://neo4j.org

http://www.sparsity-technologies.com/dex.php

http://www.openrdf.org

http://www.orientechnologies.com/orient-db.htm

https://github.com/tinkerpop/blueprints/wiki/TinkerGraph

http://rexster.tinkerpop.com

~$ cd /tmp/graphdb

graphdb$ gremlin

\,,,/ (o o)-----o00o(_)-o00o-----gremlin>

The Basics

'Gremlin is a Groovy DSL and as such, has native JVM access.'

Gremlin is a Groovy DSL and as such, has native JVM access.

1+2

3

'gremlin'

gremlin

'gremlin'.substring(2)

emlin

list = [1,2,3,4,5]

[1,2,3,4,5]

map = [key1:'value',key2:123]

key1=valuekey2=123

socket = new Socket('127.0.0.1',8182)

<modem sound/>

TextAnalysis.isMean('You are that which is not good.')

true

Graph Traversing

/tmp/graphdb

g = new Neo4jGraph('/tmp/graphdb')

/tmp/graphdb

g

g.v(1)

1

g.v(1).name

1

name=marko

g.v(1)

1

g.v(1).city

1

city=santafe

g.v(1)

1

g.v(1).map()

1

name=markocity=santafe

g.v(1)

1

g.v(1).outE

knows

knows

created

1

g.v(1).outE.since

knows

knows

created

1since=2010

since=2005

null

g.v(1).outE

knows

knows

created

1

g.v(1).outE.inV

1 knows

knows

created

2

3

4

g.v(1).outE.inV.name

1 knows

knows

created

2

3

4

name=rexster

name=stephen

name=peter

/tmp/graphdb

g

g.v(1)

1

g.v(1).outE

1 knows

knows

created

g.v(1).outE[[label:'knows']]

1 knows

knows

created

g.v(1)

1

g.v(1).outE

1 knows

knows

created

g.v(1).outE.filter{it.label=='knows'}

1 knows

knows

created

g.v(1).outE.filter{it.label=='knows'}.count()

2

g.v(1).outE

1 knows

knows

created

g.v(1).outE.filter{it.label=='knows' & since > 2006}

1 knows

knows

created

g.v(1).outE[[label:'knows']].inV.toString()

[OutEdgesPipe, LabelFilterPipe(NOT_EQUAL,knows), InVertexPipe]

g.v(1)

1

g.v(1).outE('knows')

1 knows

knows

g.v(1).outE('knows').inV

1 knows

knows

3

4

g.v(1).outE('knows').inV.toString()

[OutEdgesPipe(knows), InVertexPipe]

g.v(1)

1

g.v(1).out('knows')

1 knows

knows

3

4

g.v(1).out('knows').name

1 knows

knows

3

4

name=stephen

name=peter

g.v(1).out('knows').name.filter{it.startsWith('st')}

1 knows

knows

3

4

name=stephen

name=peter

g.v(1).out('knows').name.filter{it.matches('p.{4}')}

1 knows

knows

3

4

name=stephen

name=peter

g.v(1).out('knows').name.filter{it.matches('p.{4}') & false}

1 knows

knows

3

4

name=stephen

name=peter

g.v(1).out('knows').toString()

[OutPipe(knows)]

g.v(1)

1

g.v(1).filter{new Random().nextBoolean()}

1

WooHoo

!

g.v(1)

1


1

Alright!

g.v(1)

1


1

Doh!

Graph Inference

g.v(1)

1

g.v(1).out('created')

1

created

2

g.v(1).out('created').name

1

created

2 name=rexster

g.v(1)

1


1

created

2

g.v(1).out('created').in('created')

1

created

2

3

created

g.v(1).out('created').in('created').except([g.v(1)])

1

created

2

3

created

g.v(1).out('created').in('created').except([g.v(1)]).name

1

created

2

3

created

name=stephen

g.v(1).out('created').in('created').except([g.v(1)]).name.back(1)

1

created

2

3

created

name=stephen

g.v(1).out('created').in('created').except([g.v(1)]).name.back(1). sideEffect{g.addEdge(g.v(1),it,'collaborator')}

1 collaborator

created

2

3

created

name=stephen

g.v(1).out('created').in('created').except([g.v(1)]).name.back(1). sideEffect{g.addEdge(g.v(1),it,'collaborator')}.filter{false}

1 collaborator

created

2

3

created

name=stephen

Graph Backtracking and Looping

g.v(1)

1


1

created

2

g.v(1).out('created').out('imports')

1

created

2 imports 6

imports

5

imports

7

g.v(1).out('created').out('imports').name

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

g.v(1).out('created').out('imports').name.back(1)

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

g.v(1).out('created').out('imports').name.back(1).out('imports')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8

g.v(1).out('created').out('imports').name.back(1).out('imports').name

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

g.v(1).out('created').out('imports').name.back(1).out('imports').name .back(1)

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

g.v(1).out('created').out('imports').name.back(1).out('imports').name .back(1).out('imports')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

g.v(1).out('created').out('imports').name.back(1).out('imports').name .back(1).out('imports').out('imports')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

g.v(1).out('created').out('imports').name.back(1).out('imports').name .back(1).out('imports').out('imports').out('imports')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

g.v(1).out('created').out('imports').loop(1){it.loops < 5}

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

2


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

3


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

4


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

m = [:]

m

m

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

g.v(1).out('created').out('import').groupCount(m).loop(2){true}

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

m


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

v[5]=1v[6]=1v[7]=1

m


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

v[5]=3v[6]=2v[7]=1v[8]=1

m


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

v[5]=4v[6]=2v[7]=1v[8]=3

m


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

v[5]=4v[6]=2v[7]=1v[8]=4

m

m = [:]

m

m

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

g.v(1).out('created').out('import').name.groupCount(m).back(2).loop(2){true}

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

m


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

blueprints=1pipes=1gremlin=1

m


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

blueprints=3pipes=2gremlin=1neo4j=1

m


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j


m

Graph Paths

g.v(1)

1

g.v(1).as('x')

1

g.v(1).as('x').in('child')

1

child

9

child

10

1

child

9

child

10

g.v(1).as('x').in('child').loop('x')

1

child

9

child

10

g.v(1).as('x').in('child').loop('x'){it.object.in.count() != 0}

childchild

1

child

9

child

10


child

childchildchild

1

child

9

child

10


child

child

child

child

childchild

1

child

9

child

10


child...

100,000,000 100,000,001

child

child

child

child...

child

childchild

1

child

9

child

10

g.v(1).as('x').in('child').loop('x'){it.object.in.count() != 0}.paths

child...

100,000,000 100,000,001

child

child

child

child

childchild

[v[1], v[9], ..., v[100,000,000]][v[1], v[10], ..., v[100,000,001]]...

child...

1

child

9

child

10

g.v(1).as('x').in('child').loop('x'){it.object.in.count() != 0}.name

child...

100,000,000 100,000,001

child

child

child

child

childchild

child...

name=angelina name=brad

1

child

9

child

10

g.v(1).as('x').in('child').loop('x'){it.object.in.count() != 0}.name.paths

child...

100,000,000 100,000,001

child

child

child

child

childchild

child...

name=angelina name=brad

[v[1], v[9], ..., v[100,000,000], angelina][v[1], v[10], ..., v[100,000,001], brad]...

1

g.v(1)

1

child child

g.v(1).inE('child')

1

child child

thinks="You suck." thinks="You're great."

g.v(1).inE('child').filter{TextAnalysis.isMean(it.thinks)}

1

child

thinks="You suck."

g.v(1).inE('child').filter{TextAnalysis.isMean(it.thinks)}

1

thinks="You suck."

g.v(1).inE('child').filter{TextAnalysis.isMean(it.thinks)}.outV

9

child

1

child

thinks="You suck."

g.v(1).inE('child').filter{TextAnalysis.isMean(it.thinks)}.outV.loop(3){true}

9

thinks="You're fat."thinks="Get a real job."

thinks="You throw like a girl."

thinks="No pony!"

Graph Pattern Matching

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

SELECT ?x ?y WHERE { 1 created ?x . ?x imports ?y}

http://www.w3.org/TR/rdf-sparql-query/

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j


v[2] v[5]v[2] v[6]v[2] v[7]

?x ?y

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t = new Table()

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

g.v(1)

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

g.v(1).out('created').as('x')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

x

g.v(1).out('created').as('x').out('imports')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

x

g.v(1).out('created').as('x').out('imports').as('y')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

x

y

y

y

g.v(1).out('created').as('x').out('imports').as('y').table(t)

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

x

y

y

y

tv[2] v[5]v[2] v[6]v[2] v[7]

x y

t.get(0,1)

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

x

y

y

y

tv[2] v[5]v[2] v[6]v[2] v[7]

x y

v[5]

t.get(0,'x')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

x

y

y

y

tv[2] v[5]v[2] v[6]v[2] v[7]

x y

v[2]

t.getColumn('y')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

x

y

y

y

tv[2] v[5]v[2] v[6]v[2] v[7]

x y

v[5]v[6]v[7]

t.getColumnNames()

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

x

y

y

y

tv[2] v[5]v[2] v[6]v[2] v[7]

x y

[x, y]

t = new Table()

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

g.v(1)

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t


1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

g.v(1).out('created').as('x')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

x

g.v(1).out('created').as('x').out('imports')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

x

g.v(1).out('created').as('x').out('imports').as('y')

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

t

x

y

y

y

g.v(1).out('created').as('x').out('imports').as('y').table(t){it.id}{it.name}

t2 blueprints2 pipes2 gremlin

x y

1

created

2 imports 6

imports

5

imports

7 name=gremlin

name=blueprints

name=pipes

importsimports

imports

imports

8name=neo4j

x

y

y

y

g.clear()

Graph Construction

a = g.addVertex()

1

a.name = 'pierre'

1

name=pierre

a.city = 'brussels'

1

name=pierrecity=brussels

b = g.addVertex([name:'stephen',city:'dc'])

1


2

name=stephencity=dc

c = g.addVertex([name:'rexster',logo:[a12f04b312bc...]])

1


2

name=stephencity=dc

3

name=rexsterlogo= . reXster

g.addEdge(b,c,'created',[since:2011])

1


2

name=stephencity=dc

3

created

since=2011


g.addEdge(a,c,'reviews',[since:2011])

1

2

name=stephencity=dc

3

created

since=2011

reviews

since=2011



g.addEdge(c,g.addVertex([name:'gremlin',logo:[ab1346f47...]),'imports')

1

2

name=stephencity=dc

3

created

since=2011

reviews

since=2011


4

importsname=rexsterlogo= . reXster

name=gremlinlogo= . Gremlin

G = (V,E)

g.addEdge(c,g.addVertex([name:'pipes',logo:[12cbef46...]]),'imports')

1

2

name=stephencity=dc

3

created

since=2011

reviews

since=2011


4

imports

5

imports


G = (V,E)


name=pipeslogo= .Pipes

g.addEdge(c,g.addVertex([name:'blueprints',logo:[f13cd1467...]]),'imports')

1

2

name=stephencity=dc

3

created

since=2011

reviews

since=2011


4

imports

5

imports

6

imports


G = (V,E)



name=blueprintslogo= .Blueprints

g.V

1

2

3

4

56

g.V.count()

6

g.V.name

pierre

rexster

stephen

gremlin

blueprintspipes

g.V

1

2

3

4

56

g.V.logo

GremlinG = (V,E)

reXster

PipesBlueprints

g.V.logo.count()

4

g.V.transform{it.map()}

name=stephencity=dc



G = (V,E)



name=blueprintslogo= .Blueprints

6

g.V.transform{it.map()}.count()

7

g.V.transform{it.map()}.count() + 1

8

g.V.transform{it.map()}.count() + 1 + 1

4

(g.V.transform{it.map()}.count() + 1 + 1) / 2

4 is a number.

((g.V.transform{it.map()}.count() + 1 + 1) / 2) + ' is a number.'

14

(((g.V.transform{it.map()}.count() + 1 + 1) / 2) + ' is a number.').length()

g.E

created

reviews

imports

importsimports

g.E.transform{it.map()}

since=2011

since=2011

g.clear()

Graph Ranking

g.loadGraphML('loopy-lou-and-the-loops.xml')

1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

m = [:]

1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

m

c = 0

1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

mm

c 0

g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

m

c 0


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[2]=1v[6]=1m

c 0


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[2]=1v[6]=1m

c 2


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[2]=1v[3]=1v[6]=2v[7]=1

m

c 2


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[2]=1v[3]=1v[6]=2v[7]=1

m

c 5


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=1v[2]=1v[3]=1v[4]=1v[6]=2v[7]=2v[8]=1

m

c 5


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=1v[2]=1v[3]=1v[4]=1v[6]=2v[7]=2v[8]=1

m

c 9


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=3v[2]=2v[3]=1v[4]=1v[5]=1v[6]=3v[7]=2v[8]=2

m

c 9


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=3v[2]=2v[3]=1v[4]=1v[5]=1v[6]=3v[7]=2v[8]=2

m

c 15


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=4v[2]=4v[3]=3v[4]=1v[5]=1v[6]=7v[7]=3v[8]=2

m

c 15


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=4v[2]=4v[3]=3v[4]=1v[5]=1v[6]=7v[7]=3v[8]=2

m

c 25


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=5v[2]=5v[3]=5v[4]=3v[5]=1v[6]=10v[7]=7v[8]=3

m

c 25


1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=5v[2]=5v[3]=5v[4]=3v[5]=1v[6]=10v[7]=7v[8]=3

m

c 39

println 'do you understand the concept of centrality?'

1

friend

2 3

4

5

67

8

friend

friend

friend

friend

friend

friend

friend friend

friend

friend

friend

v[1]=5v[2]=5v[3]=5v[4]=3v[5]=1v[6]=10v[7]=7v[8]=3

m

c 39

g.clear()

reXster

IDE Development

Command Line REPL REST Server





Via any JVM language (JSR 223)

Neo4j Server

the pathology of graph databases

Technology