S2Graph : A large-scale graph database

with Hbase

daumkakao

Doyoung Yoon x Taejin Chin

2

DaumKakaoA Mobile Lifestyle Platform

1. KakaoTalka. Mobile Messenger replacing SMSb. ‘KaTalkHe’ is being used as a verb in Korea like ‘Googling’c. 96% of Korean smartphone users are using KakaoTalkd. 170M users worldwidee. 3B messages / day

3

KakaoTalk

Social Platform

DaumKakaoA Mobile Lifestyle Platform

KakaoStory

KakaoGroup

Daum Cafe

ContentsPlatform

KakaoTopic

KakaoPage

KakaoGame

CommercePlatform

KakaoPick

KakaoMusic

MarketingPlatform

Yellow IDMedia Daum

Daum tvPot

LocalPlatform

Daum Map

Daum Webtoon

PersonalPlatform

Sol calendarKakaoPlace

Sol Mail

Zap

Plus FriendGift Shop

Digital Item Store KakaoStyle

KakaoHome

Sol Group

Story Plus

Daum Cluod

Biggest mobile SNS in Korea

96% of Korean smartphone

users are using KakaoTalk messenger, 170 million

users worldwide)

4

Our Social Graph

Message

Writelength :

Read

Couponprice :

Presentprice : 3

affinity affinity:

affinity

affinity

affinity

affinity

affinity

affinity

affinity

Friend

Groupsize : 6

Emoticon

Eatrating :

Viewcount :

Playlevel: 6

PickwithFriend :

Advertise

Searchkeyword :

Listencount :

Likecount : 7

Comment

affinity

5

Our Social Graph

Messagelength : 9

Writelength : 3

affinity 6affinity: 9

affinity 3

affinity 3

affinity 4

affinity 1

affinity 2

affinity 2

affinity 9

Friend

Playlevel: 6

PickwithFriend : 3

Advertisectr : 0.32

Searchkeyword : “HBase"

Listencount : 6

Commentlength : 15

affinity 3

Message ID : 201

Ad ID : 603Music ID : 603

Item ID : 13

Post ID : 97

Game ID : 1984

6

Technical Challenges

1. Large social graph constantly changing

a. Scale

more than,social network: 10 billion edges, 200 million vertices, 50 million update on existing edges.user activities: 400 million new edges per day

7

Technical Challenges (cont)

2. Low latency for breadth first search traversal on connected data.

a. performance requirement

peak graph-traversing query per second: 20000response time: 100ms

8

Technical Challenges (cont)

3. Update should be applied to graph in real time for viral effect

Person A

PostFast Person B

Comment

Person C

Sharing

Person D

MentionFast Fast

9

Technical Challenges (cont)

4. Support for Dynamic Ranking logic

a. push strategy: hard to change data ranking logic dynamically.

b. pull strategy: can try various data ranking logic

10

Before

Each app server should know each DB’s sharding logic. Highly inter-connected architecture

Friend relationship SNS feeds Blog user activities Messaging

Messaging App

SNS App

Blog App

11

After

SNS App

Blog App

Messaging App

S2Graph DBstateless app servers

12

S2Graph : Distributed Online GraphDB

1.Low-latency

2.Graph-traversable

3.Scalable

4.Eventually consistent

5.Asynchronous, non-blocking

13

Why We Choose HBase?

1.High Availability

2.Scalability

3.Low latency

4.High concurrency

5.Fault tolerant

6.Integration with HDFS

7.Distributed operation

14

The Data Model

1. Columns2. Labels3. Directions4. Index Properties5. Non-index Properties

1 3comment

4

know created

name = “josh”age = 32

edge 2 source vertex

vertex 1 out edges

edge 2 target vertex

2

edge 2 label

vertex 2 in edges

vertex 4 id

vertex 4 properties

date = 20150507

edge 5 properties

5

15

How to store the data - EdgeLogical View

1. Snapshot edges : Up-to-date status of edge

columnrow

Tgt Vertex ID1 Tgt Vertex ID2 Tgt Vertex ID3

Src Vertex ID1 Properties Properties Properties

Src Vertex ID2 Properties Properties Properties

a. Fetching an edge between two specific vertexb. Lookup Table to reach indexed edges for update, increment, delete operations

16

How to store the data - EdgeLogical View

2. Indexed edges : Edges with index

column

rowIndex Values | Tgt Vertex ID1 Index Values | Tgt Vertex ID2

Src Vertex ID1 Non-index Properties Non-index Properties

a. Fetches edges originating from a certain vertex in order of index

17

How to store the data - EdgePhysical View - table schema

1. Snapshot Edgea. Rowkey

Murmur Hash Src Vertex ID Label ID Direction Index Sequence Is Inverted

16 bit variable length 30 bit 2 bit 7bit 1 bit

Vertex IDs can be encoded with 8 bit header + byte array (long, integer, short, byte, string)

18

How to store the data - EdgePhysical View - table schema

1. Snapshot Edgeb. Qualifier

Target Vertex ID

variable length

c. Value

All Property Key Value Pairs

variable length

19

How to store the data - EdgePhysical View - table schema

2. Indexed Edgea. Rowkey

Murmur Hash Src Vertex ID Label ID Direction Index Sequence Is Inverted

16 bit variable length 30 bit 2 bit 7bit 1 bit

Vertex IDs can be encoded with 8 bit header + byte array (long, integer, short, byte, string)

20

How to store the data - EdgePhysical View - table schema

2. Indexed Edgeb. Qualifier

Index Property Values Tgt Vertex ID

variable length variable length

c. Value

Non-index Property Key Value Pairs

variable length

21

How to store the data - VertexLogical View

1. Vertex : Up-to-date status of Vertex

columnrow

Property Key1 Property Key2

Src Vertex ID1 Value1 Value2

Vertex ID2 Value1 Value2

22

How to store the data - VertexPhysical View - table schema

1. Vertex : Up-to-date status of Vertex

a. Rowkey

Murmur Hash Column ID Vertex ID

16 bit integer(32bit) variable length

b. Qualifier

Property Key

Byte(8 bit)

c. Value

Property Value

variable length

23

How to read the data - GetEdges

Using a custom query DSL on top of HTTP

curl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{

"srcVertices": [{"serviceName": "s2graph", "columnName": "account_id", "id":1}],

"steps": [

[{"label": "friends", "direction": "out", "limit": 100}], // step

[{"label": "hear", "direction": "out", "limit": 10}]

]

}

'

Steps = a list of StepStep = contains the labels to traverse and how to rank them in the result

Step 1

friend 1

heartime: 20140502

heartime: 20140712

heartime: 20141116

Friends Friends

friend 2

User 1

Step 2

Don’t let go let it be let it go

24

How to read the data - GetEdges Example

Friend list

curl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{

"srcVertices": [{"serviceName": "s2graph", "columnName": "account_id", "id":1}],

"steps": [

[{"label": "friends", "direction": "out", "limit": 100}], // step

]

}

'

friend 1 friend 2

User 1

Friends Friends

25

How to read the data - GetEdges Example

Songs my friends have listened

curl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{

"srcVertices": [{"serviceName": "s2graph", "columnName": "account_id", "id":1}],

"steps": [

[{"label": "friends", "direction": "out", "limit": 50, “scoring”: {“score”: 1.0}],

[{"label": "listen", "direction": "out", "limit": 10}]

]

}

'

friend 1

Friends Friends

friend 2

Don’t let go let it be let it go

heartime: 20140502

heartime: 20140712

heartime: 20141116

User 1

Reference : https://github.com/daumkakao/s2graph#1-definition

26

How to read the data - GetEdges Example

Similar songs to songs that I have listened to.

curl -XPOST localhost:9000/graphs/getEdges -H 'Content-Type: Application/json' -d '

{

"srcVertices": [{"serviceName": "s2graph", "columnName": "account_id", "id":1}],

"steps": [

[{"label": "listen", "direction": "out", "limit": 50}],

[{"label": "similar_song", "direction": "out", "limit": 10, “scoring”: {“score”: 1.0}]

]

}

User 1

Don’t let go let it be let it go

heartime: 20140502

heartime: 20140712

heartime: 20141116

let it bleed Hey jude Do you wannabuild a snowman?

similar_songsimilarity: 0.3

similar_songsimilarity: 0.4

similar_songsimilarity: 0.6

27

How to read the data - GetVertices

curl -XPOST localhost:9000/graphs/getVertices -H 'Content-Type: Application/json' -d '

[

{"serviceName": "s2graph", "columnName": "account_id", "ids": [1, 2, 3]},

{"serviceName": "kakaomusic", "columnName": "user_id", "ids": [1, 2, 3]}

]

'

User 1{created_at:20070812,updated_at:20150507}

User 2{created_at:201206132,updated_at:20140505}

28

How to write the data - Insert

curl -XPOST localhost:9000/graphs/edges/insert -H 'Content-Type: Application/json' -d '

[

{"from":1,"to":2,"label":"graph_test","props":{"time":-1, "weight":10},"timestamp":1417616431},

]

'

User 1 User 2

29

How to write the data - Delete

curl -XPOST localhost:9000/graphs/edges/delete -H 'Content-Type: Application/json' -d '

[

{"from":1,"to":2,"label":"graph_test","timestamp":1417616431},

{"from":1,"to":3,"label":"graph_test","timestamp":1417616431},

]

'

User 1 User 2

30

How to write the data - Update

curl -XPOST localhost:9000/graphs/edges/update -H 'Content-Type: Application/json' -d '

[

{"from":1,"to":2,"label":"graph_test","timestamp":1417616431, "props": {"is_hidden": true, “status”: 200},

{"from":1,"to":3,"label":"graph_test","timestamp":1417616431, "props": {"status": -500}

]

User 1 User 2

friend{is_hidden:true,

status:200}

31

REST API Spec.

Read

1. getEdges2. checkEdge3. getEdgesCount4. getVertices

Write

1. insert2. delete3. update4. increment

Management

1. create service (vertex type)2. create label (edge type)3. add Index

32

HBase Table Configuration

1. setDurability(Durability.ASYNC_WAL)

2. setCompressionType(Compression.Algorithm.LZ4)

3. setBloomFilterType(BloomType.Row)

4. setDataBlockEncoding(DataBlockEncoding.FAST_DIFF)

5. setBlockSize(32768)

6. setBlockCacheEnabled(true)

7. pre-split by (Intger.MaxValue / regionCount). regionCount = 120 when create table(on 20 region server).

33

HBase Cluster Configuration

• each machine: 8core, 32G memory

• hfile.block.cache.size: 0.6

• hbase.hregion.memstore.flush.size: 128MB

• otherwise use default value from CDH 5.3.1

34

Overall Architecture

S2graphClients

35

Compare to other Online GraphDBs

Titan (v0.4.2)

a. Pros- Rich API and easy to setup- Relatively large community- Transaction handling

b. Cons- Using it’s own ID system; less efficient for graph traversal (details in next slide)- Index data stored on one region (hotspot) with strong consistency option- Not many references on Titan with HBase comparing to other storages

36

Compare to Titan

Titan is less efficient for graph traversal - For following 1 normal graph traversal query, Vertex(“userID:1”).out(“friends”).limit(10).out(“friends”).limit(10) User 1

friends

friends

37

Compare to Titan (cont)Vertex(“userID:1”).out(“friend”).limit(10).out(“friend”).limit(10)

Titan S2graph

# of read requests on HBase

112 = 1 (Vertex Lookup : a) + 1 (1st step edges : b) + 10 (2nd step edges : c) + 100 (Destination Vertices : d)

11 = 1 (1step edges : e) + 10 (2nd step edges : f)

Titan S2graphB

A

C

D

e

f

38

Performance

1. Test dataa. Total # of Edges: 9,000,000,000b. Average # of adjacent edges per vertex: 500c. Seed vertex: vertices that has more than 100 adjacent edges.

2. Test environmenta. Zookeeper server: 3b. HBase Masterserver: 2c. HBase Regionserver: 20d. App server: 8 core, 16GB Ram

39

- Benchmark Query : src.out(“friend”).limit(50).out(“friend”).limit(10)- Total concurrency: 20 * # of app server

Performance

2. Linear scalability

Late

ncy

0

50

100

150

200

QPS

0

1,000

2,000

3,000

4,000

# of app server1 2 4 8

QPS(Query Per Second) Latency(ms)

51515047

3,097

1,567803

42147 50 51 51

# of app server1 2 3 4 5 6 7 8

500

1000

1500

2000

2500

3000

QPS

40

Performance

3. Varying width of traverse (tested with a single server)

Late

ncy

0

75

150

225

300

QPS

0

500

1,000

1,500

2,000

Limit on first step20 40 80 100

QPS Latency(ms)

97754323 203266457

943

23 4375 97

- Benchmark Query : src.out(“friend”).limit(x).out(“friend”).limit(10)- Total concurrency = 20 * 1(# of app server)

Performance

3. Varying width of traverse (tested with a single server)

Late

ncy

0

75

150

225

300

QPS

0

500

1,000

1,500

2,000

Limit on first step20 40 80 100

QPS Latency(ms)

97754323 203266457

943

23 4375 97

- Benchmark Query : src.out(“friend”).limit(x).out(“friend”).limit(10)- Total concurrency = 20 * 1(# of app server)

42

- All query touch 1000 edges.- each step` limit is on x axis.- Can expect performance with given query`s search space.

Performance

4. Different query path(different I/O pattern)

Late

ncy

0

37.5

75

112.5

150

QPS

0

80

160

240

320

400

limits on path10 -> 100 100 -> 10 10 -> 10 -> 10 2 -> 5 -> 10 -> 10 2 -> 5 -> 2 -> 5 -> 10

QPS Latency(ms)

5667716867

352.2298.1280272.5297

67 68 71 67 56

43

Performance

5. Write throughput per operation on single app server

Insert operation

Late

ncy

0

1.25

2.5

3.75

5

Request per second

8000 16000 800000

44

Performance

6. write throughput per operation on single app server

Update(increment/update/delete) operation

Late

ncy

0

2

4

6

8

Request per second

2000 4000 6000

45

Stats

1. HBase cluster per IDC (2 IDC)- 3 Zookeeper Server- 2 HBase Master- 20 HBase Slave

2. App server per IDC- 10 server for write-only- 20 server for query only

3. Real traffic- read: over 10K request per second

- now mostly 2 step queries with limit 100 on first step.- write: over 5k request per second

* Deep traversal queries are not counted since it is in test stage for production

46

47

Through HBase !

48

Now Available As an Open Source- https://github.com/daumkakao/s2graph- Finding a mentor

Contact- Taejin Chin : taejin.chin@gmail.com- Doyoung Yoon : shom83@gmail.com

49

Late

ncy

0

50

100

150

200

QPS

0

500

1,000

1,500

2,000

# of app server1 2 3 4 5

Native Client QPSNative Client Latency(ms)

174186189177178

570429315224112

178 177 189 186 174

- Benchmark Query : src.out(“friend”).limit(50).out(“friend”).limit(10)- Test seed edges have adjacent edges more than 100: 30millions- Total concurrency: 20 * # of app server

Appendix

Late

ncy

0

50

100

150

200

QPS

0

500

1,000

1,500

2,000

# of app server1 2 3 4 5

Asnychbase QPSAsynchbase Latency(ms)

5351505047

1,8951,567

1,192803

42147 50 50 51 53

3.5x performance improvement using Asynchbase