trivadis techevent 2016 introduction to datastax enterprise (dse) graph by guido schmutz

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH

Introduction to DataStax Enterprise (DSE) GraphGuido Schmutz

@gschmutz guidoschmutz.wordpress.com

Agenda

1. Why a Graph Database?

2. DataStax Enterprise Edition – DSE Graph

3. DSE Graph in Action• Working with Graph Schemas• Inserting Graph Data• Traversing Graph Data• Indexing Graph Data

4. Summary

Introduction to DataStax Enterprise (DSE) Graph2 9/9/2016

Why a Graph Database?


Know your domain

ConnectednessofDatalow high

DocumentDataStore

Key-ValueStores

Wide-ColumnStore

GraphDatabases

RelationalDatabases


RDBMS vs. Graph Database


RDBMS GraphDatabase

Processtoquerydataelements(joins)isinefficientonlargedatasetormanyrelationships

Betterperformanceforrelationshipqueriesdue tospecializedindexstructures

ExpressingJOIN-intensivequeriesinSQL istime-consuminganderrorprone

Intuitivequerylanguageenablingfasterapplicationdevelopment

RDBMS vs. Graph Database – Data Modelling


VS. !"

!"

!"

#" #"

$"

$"

$"

partner!

worksFor !

develops!

develops!

uses!

uses!

reportsTo!

reportsTo!

manages!

worksFor !

worksFor !

Source:DataStax

Graph Use Cases


Master Data Management

Customer 360

Recommendation

Personalization

Security

Fraud Detection

Internet of Things (IoT)

Introduction to the Graph Model – Property Graph

Node / Vertex• Represent Entities• Can contain properties

(key-value pairs)

Relationship / Edge• Lines between nodes • may be directed or

undirected

Properties• Values about node or

relationship• Allow to add semantic to

relationships

User1 Movierated

knows rated

User2

userId:16134540name:Peterlocation:PaloAlto

userId:18898576name:Guidolocation:Berne

rating:8

key:value


rating:6movieId:m1000title:Sullyyear:2016duration:95country:UnitedStatesproduction:[FilmNation,

FlashlightFilms]

DataStax Enterprise Edition – DSE Graph


DataStax Enterprise Edition – DSE Graph

Introduction to DataStax Enterprise (DSE) Graph14

• A scale-out graph database purposely built for cloud applications that need to act on complex and highly connected relationships.

• Supports a property graph model natively inside the DataStax product, engineered specifically for DataStax Enterprise (Cassandra, Search, Analytics).

• Store & find relationships in data fast and easy in large graphs.

• Part of DSE’s multi-model platform.

9/9/2016

Image:DataStax

DataStax Enterprise Graph


Real-time graph database management system

• Adopts Apache TinkerPop standards for data and traversal

• Uses Apache Cassandra for scalable storage and retrieval

• Leverages Apache Solr for full-text search and indexing

• Integrates Apache Spark for fast analytic traversal

• Supports comprehensive data security for the enterprise

Image:DataStax

Why Cassandra?

• All nodes participate in a cluster• Shared nothing• Add or remove as needed• More capacity? Add more servers

• Each node owns a range of partitions• Consistent Hashing

DSE Graph - Architecture


Image:DataStax

Today’s Needs and the solution in DSE Graph

Need DSEGraphFeature

StoreandAccessDataQuickly GraphDataModel+Gremlin

Flexible,FastApplicationBuilds DSEGraph+DSEStudio &DSEDrivers

AnalyzeInformation GraphAnalyticswithDSEAnalytics

SearchandFindQuickly GraphSearchwithDSESearch

Ingestand ExportData DSEGraphLoader

SecureInformation DSESecurity

ManageandMonitor Opscenter

Availability,Scale,OperationalEase,Performance…

DataStax Enterprise


• Web-based developer solution which helps developers visually explore, query, and trouble-shoot DSE Graph in one intuitive UI.

• Auto-completion, result set visualization, execution management, and much more.

Developer support with DataStax Studio


Data Loading Support with DSE Graph Loader

• Simplifies loading large amounts of enterprise data from various sources into DSE Graph efficiently and robustly.

• Inspects incoming data for schema compliance.• Uses declarative data mappings and custom transformations to handle diverse types

of data.

Graph Loader

� ��Data Mappings

Batch Loading

Stream Ingestion

RDBMSJSON

DSE Graph


Operational support with DataStax OpsCenter

• Web-based operations solution which can launch, manage, monitor and trouble-shoot DSE clusters and deployments.

• Launch wizard, failure alerts, monitoring dashboards, and much more.

DSE Graph in Action


Sample Graph: Movies Database


Person

Genre

movieId : texttitle : textyear : intduration : intcountry : textproduction : text*

personId : textname : text

User Movie belongsTo

actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int

userId : textage : intgender : text

genreId : intname : text

DSE Graph in Action – Working with Graph Schemas


Graph Schema API


Schema management basics• Accessible via schema object• Defined for each graph in the DSE Graph system• Provides methods propertyKey, vertexLabel, edgeLabel, etc.• Supports fluent interface

Four important elements of a graph schema• Property keys• Vertex labels• Edge labels• Indexes

Graph Schema API – Property Keys


Define types of properties to be used with Vertices and Edges

• Creating single-cardinality property keys (single() is assumed by default)

• Creating multi-property keys (can only be associated with vertices)

• Creating meta-property keys (can only be associated with vertices)

schema.propertyKey("title").Text().single().create(); schema.propertyKey("year").Int().create();

schema.propertyKey("production").Text().multiple().create();

schema.propertyKey("source").Text().create();schema.propertyKey("date").Timestamp().create();schema.propertyKey("budget").Text().multiple().

properties("source","date").create();

Movie

movieId: Texttitle: Textyear: Intduration: Intcountry: Textproduction: Text*

Graph Schema API – Vertex Labels


Define types of vertices to be used in a graph

• Creating vertex labels

• Different labels can use same property key

schema.vertexLabel("movie").properties("movieId","title",”year",”duration",”country","production").create();

schema.vertexLabel("genre").properties("genreId","name").create(); schema.vertexLabel("person").properties("personId","name").create();

Movie

movieId: Texttitle: Textyear: Intduration: Intcountry: Textproduction: Text*

Genre

genreId:Textname:Text

Person

personId:Textname:Text

Graph Schema API – Edge Labels


Define types of edges to be used in a graph

• Creating single-cardinality edge label definition

• Creating multi-cardinality edge label definition (multiple() is assumed by default)

schema.edgeLabel("rated").single().properties("rating").connection("user","movie").create();

schema.edgeLabel("actor").multiple().properties("rating").connection("movie","person").create();

User Movierated(0..1)

rating:int

Person Movieactor(0..*)

Graph Schema API – Management of Graph Schemas


• Listing all schema definitions

• Listing individual schema elements

• Dropping all schema definitions (will also delete all graph data !!!)

schema.describe()

schema.vertexLabel("user").describe()schema.edgeLabel("rated").describe()

schema.clear()

DSE Graph in Action – Inserting Graph Data


Creating Vertices and Vertex Properties


Adding a new User vertex

Adding a new Movie vertex

Vertex u = graph.addVertex("user"); u.property("userId","u2016"); u.property("age",36); u.property("gender","M");

UseruserId:u2016age:26gender:M

Vertex m = graph.addVertex(label, "movie","movieId", "m1000","title", "Sully","year", 2016, "duration", 95, "country", "United States");

m.property("production", "FilmNation"); m.property("production", "Flashlight Films");

MoviemovieId:m1000title:Sullyyear:2016Duration:95Country:UnitedStatesProduction:[FilmNation,

FlashlightFilms]

Creating Vertices and Vertex Properties


Adding an edge between the user vertex and the movie vertex

Vertex u = graph.addVertex("user", ...); Vertex m = graph.addVertex("movie", ...);

Edge e = u.addEdge("rated", m);e.property("rating", 8);

UseruserId:u2016age:26gender:M

Movie

rated

rating:8

movieId:m1000title:Sullyyear:2016Duration:95Country:UnitedStatesProduction:[FilmNation,

FlashlightFilms]

Gremlin I/O


Exporting and importing existing graphs and subgraphs• Migration between Apache TinkerPop-enabled graph databases• Serialization and interchange of graphs between databases and tools• Creating backup copies of graph• Different file formats for graph serialization

Format Description

GraphML CommonXMLvocabularyforgraphrepresentation. Supportedbymanygraph-relatedtools,applicationsandlibraries."Lossy" formatasitlackssupportforcomplexdatatypesandgraphvariables.

GraphSON Apache TinkerPop-originatedJSON-based,losslessformatforgraphrepresentation.

Gryo Fast,space-efficient, losseless,binarygraphserializationformatforusebyJVMlanguages.Gryo isenalbed byKryo – afastandefficientobjectgraphserializationframework.

Gremlin I/O


Exporting to and importing from a GraphML file

Exporting to and importing from a GraphSON file

Exporting to and importing from a Kryo file

graph.io(IoCore.graphml()).writeGraph("KillrVideo.xml")

graph.io(IoCore.graphml()).readGraph("KillrVideo.xml")

graph.io(IoCore.graphson()).writeGraph("KillrVideo.json")

graph.io(IoCore.graphson()).readGraph("KillrVideo.json")

graph.io(IoCore.gryo()).writeGraph("KillrVideo.json")

graph.io(IoCore.gryo()).readGraph("KillrVideo.json")

DSE Graph in Action – Traversing Graph Data


Gremlin Traversals in DSE Graph


• Standard Gremlin Language• Gremlin is defined by Apache TinkerPop• Expressive fluent language to define traversals (Groovy)

• Production vs. Development modes• Production (default) requires an explicitly defined graph schema and proper graph indexes to

avoid expensive scans

• Enabling graph scans in production mode (acceptable if scanning small portions of data => caution is advised!)

g.V().has("movie","movieId","m366").values("title","year")

schema.config().option("graph.schema_mode").get()schema.config().option("graph.schema_mode").set("Development")

schema.config().option("graph.allow_scan").set(true)

Gremlin Traversals in DSE Graph


OLTP vs. OLAP Traversals

OLTPtraversals OLAPtraversals

Generateinstantaneous responses Take longertoexecute

Resembletargeteddatabasequeries Involve broader-scopedataanalysis

Heavily relyonvertexidsorindexes Require expensivegraphscans

Accesssmallsubgraphs Moveto theoutgoingedges

Traverseshortpathswithfewbranches Movetothe incomingedges

Gremlin Traversal Ingredients


Traversal source

Traversal Steps

Traverser

g = grap.traversal()

g.V().has("title","Alice in Wonderland").has("year",2010).out("director").values("name")

Defining Gremlin Traversals


Linear motif: Traversal is a sequence of steps

g. V().

has("title","Alice in Wonderland"). has("year",2010). out("director"). values("name")

// Sample Output:// Tim Burton

Person

Movie

screenwriter

movieId:m267title:AliceinWonderlandyear:2010…

Person

Person

actor

personId:p5206Name:LindaWoolverton

director

personId:p8153Name:TimBurton

personId:p4361Name:JohnnyDepp

Defining Gremlin Traversals


Netsted motif: Traversal is a tree of steps

g. V().

has("title","Alice in Wonderland"). has("year",2010). union(__.out("director"),

out("screenwriter")). values("name")

// Sample Output:// Tim Burton// Linda Woolverton

Person

Movie

screenwriter

movieId:m267title:AliceinWonderlandyear:2010…

Person

Person

actor

personId:p5206Name:LindaWoolverton

director

personId:p8153Name:TimBurton

personId:p4361Name:JohnnyDepp

Simple Traversal


Navigating from a Vertex:

Step Description

out([label],…) Movetotheoutgoing vertices

in([label],…) Movetotheincomingvertices

both([label],…) Movetoboththeincomingandoutgoingvertices

outE([label],…) Moveto theoutgoingedges

inE([label],…) Movetothe incomingedges

bothE([label],…) Movetoboththeincomingandoutgoingedges

MovieUser

Genre

Genre

rated

belongsTo

actor

both()bothE()

in() inE() outE() out()

Simple Traversal


Navigating from an Edge:

Step Description

outV() Movetotheoutgoing vertices

inV() Movetotheincomingvertices

bothV() Movetoboththeincomingandoutgoingvertices

otherV() Moveto thevertexthatwasnotthevertexthatwasmovedfrom

MovieUserrated

outV() bothV() inV()

Simple Traversal with in


Find Jonny Depp’s movies release in 2010 or later:

g.V().hasLabel("person"). has("name","Johnny Depp"). in("actor"). has("year",gte(2010)). values("title")

"Into the Woods","Pirates of the Caribbean: On Stranger Tides","Alice in Wonderland"

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Simple Traversal with in and inE


Find user rating for Jonny Depp’s movies released in 2010 or later:

g.V().hasLabel("person"). has("name","Johnny Depp"). in("actor"). has("year",gte(2010)). inE("rated").values("rating")

3,7,7,7,5,5,8,...

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Simple Traversal with in, inE and outV


Find ages of users who left 7 or 8 star rating for Johnny Depp’s movies released in 2010 or later:

g.V().hasLabel("person"). has("name","Johnny Depp"). in("actor"). has("year",gte(2010)). inE("rated").has("rating", within(7,8)).outV().values("age")

60,37,63,62,57,...

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Path Traversal with path


Find a path from a user to a friend

g.V().has("user","userId","u1"). out("knows").out("knows").path().by("userId").limit(1)

g.V().has(“user“,"userId","u1"). out(“knows“).out("knows").path().by("age").limit(1)

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Path Traversal with path


Find a shortest, simple path between to actors

g.V().has("person","name","Johnny Depp"). repeat(both("actor").simplePath().timeLimit(1000)).until(has("person","name","Leonardo DiCaprio")). path().by(choose(hasLabel("person"),

values("name"),values("title"))).limit(1)

// Sample Output// "Johnny Depp", "Dead Man", "Gabriel Byrne", // "The Man in the Iron Mask", "Leonardo DiCaprio"

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Projecting Traversal with select


Find Johnny Depp’s movie titles

g.V().has("person","name","Johnny Depp").as("actor"). in("actor").as("movie"). select("actor","movie"). by("name"). by("title").sample(1)

// Sample Output// "actor": "Johnny Depp", "movie": "Dead Man"

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Projecting Traversal with select


Find Johnny Depp’s movie titles, years, and genres

g.V().has("person","name","Johnny Depp").as("a"). in("actor").as("t","y").out("belongsTo").as("g"). select("a","t","y","g"). by("name").by("title").by("year").by("name").sample(3)

// Sample output: // [a:Johnny Depp, t:Alice in Wonderland, y:2010, g:Fantasy] // [a:Johnny Depp, t:Alice in Wonderland, y:2010, g:Animation] // [a:Johnny Depp, t:Alice in Wonderland, y:2010, g:Adventure]

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Statistical Traversal with count


Find the number of vertices and edges in a graph

g.V().count()

// 10797

g.E().count()

// 69054

g.V().hasLabel("user").count()

// 1100

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Statistical Traversal with mean, min and max


Find the average, smallest and largest user ages

g.V().hasLabel("user").values("age").mean()

// 32.01282051282051

g.V().hasLabel("user").values("age").min()

// 12

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Statistical Traversal with group().by().by() pattern


Find the average ages of female and male users in the graph

g.V().hasLabel("user").group().by("gender").by(values("age").mean())

// F : 30.561371841155236// M : 32.01282051282051

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



Statistical Traversal with groupCount


Find the vertex and edge distribution by label and graph

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



g.V().groupCount().by(label)

// [movie:920, // person:8759, // genre:18, // user:1100]

Declarative Traversal with match


Find directors who appeared in their own movies

g.V().match(__.as("m").out("actor").as("p"), __.as("m").out("director").as("p")).

select("p").by("name").dedup(). sample(3)

// Sample Output.// "Gene Kelly",// "Sylvester Stallone",// "Terry Jones"

Person

Genre




actor

rated

director

composer

screenwriter

cinematographer

knows

rating : int



DSE Graph in Action – Indexing Graph Data


Graph Schema API – Vertex Indexes (I)


Efficiently retrieve all vertices with a known label and a given vertex value

• Creating and Using materialized view index on a high cardinality property

• Creating and Using secondary index on a low cardinality property

schema.vertexLabel("movie").index("moviedsById").materialized().by("movieId").add()

g.V().hasLabel("movie").has("movieId","m267")

schema.vertexLabel("movie").index("moviesByYear").secondary().by("year").add()

g.V().has("movie","year",2010)

Graph Schema API – Vertex Indexes (II)


Efficiently retrieve all vertices with a known label and a given vertex value

• Creating and Using full text search index on a text property

• Creating and Using string search index on a text property

schema.vertexLabel("movie").index("search").search().by("title").asText().add()

g.V().has("movie","title",Search.tokenRegex("Wonder.*"))

schema.vertexLabel("movie").index("search").search().by("country").asString().add()

g.V().has("movie","country",Search.prefix("U"))

Graph Schema API – Property Indexes


Efficiently retrieve properties of a known vertex that have associated meta-properties whose values are known or fall into a known range

• Creating and Using a Property Index

schema.vertexLabel("movie").index("movieBudgetBySource").property("budget").by("source").add()

g.V().has("movie","movieId","m267").properties("budget")..has("source","Los Angeles Times").value()

Movie

movieId: Texttitle: Textyear: Intduration: Intcountry: Textproduction: Text*budget: Int*[

source:Textdate:Timestamp]

schema.vertexLabel("movie").index("movieBudgetByDate").property("budget").by("date").add()

Graph Schema API – Edge Indexes (I)


Efficiently traverse edges that are incident to a known vertex, have a known label, and have properties whose values are known or fall into a known range

• Creating and Using an Edge Index

• Find how many users rated a particular movie with an8-star rating

schema.vertexLabel("movie").index("toUsersByRating").inE("rated").by("rating").add()

g.V().has("movie", "movieId", "m267").inE("rated").has("rating",8).count()

movieId:Text…


rating:int

Graph Schema API – Edge Indexes (II)


Efficiently traverse edges that are incident to a known vertex, have a known label, and have properties whose values are known or fall into a known range

• Creating and Using an Edge Index

• Find movies rated with a greater-than-7 ratingby a particular user

schema.vertexLabel("user").index("toMoviesByRating").outE("rated").by("rating").add()

g.V().has("user", "userId", "u1").outE("rated").has("rating",gt(7)).inV()

movieId:Text…


rating:int

userId:Text…

Summary


Summary - A Complete Integrated Solution for Graph

Introduction to DataStax Enterprise (DSE) Graph63

ServerVisualManagement/Monitoring

VisualDevelopmentIntegratedDrivers(CQL,Gremlin,etc.)

JavaPythonC++More…

9/9/2016 Images:DataStax

Trivadis Enterprise Knowledge Graph


Presentation

Project

Employee

Event

Course

Term

Certification Department

Locationskill (level, since)

teaches

located in (since)

related to

related to

author

participates in (since, role)

part of (since)owns (since)

related to

related to

created for

related to / child of

managed by

Customer for

Event Type isOf

managed bylocated in

responsible unit

owned by

Project Type

isOf

Industry

forin

participates in

Publication

related to

for

Social Media Profile

uses

related to

Instant Messaging

uses

located in

responsible (for)

previous year

main teacher

Armasuisse W&T Social Network Graph


Twitter User Tweetpublishes (timestamp)

Term

mentions (timestamp)

Place

uses

followedBy

Youtube Channel

idnamelanguagetimestamplastUpdateTime

Youtube Videopublishes (timestamp)

nametypelastUpdateTime

idtargetIdstimestamplanguagelastUpdateTime

uses

idstreetcountrynametypeurllastUpdateTime

Url

idnamehasFavoriteshasLikeshasDislikeshasCommentslastUpdateTime

uses

Person

owns

Webpage

retweetsreplies

idshortUrlurllastUpdateTime

usesuses

linksTo

owns

linksTo

located

idnamelastUpdateTime

idnamelastUpdateTime

Geo PointidlocationaltitudelastUpdateTime

uses

uses

Entity

uses (time)

uses

uses


More Information

• DS330: DataStax Enterprise Graph – Self-paced course offered for free by DataStax

Guido SchmutzTechnology Manager

[email protected]

9/9/2016 Introduction to DataStax Enterprise (DSE) Graph68

@gschmutz guidoschmutz.wordpress.com