tech talk: rocksdb slides by dhruba borthakur & haobo xu of facebook

The Story of RocksDB

Dhruba Borthakur & Haobo XuDatabase Engineering@Facebook

Embedded Key-Value Store for Flash and RAM

Monday, December 9, 13

A Client-Server Architecture with disks

Network roundtrip = 50 micro sec

Disk access = 10 milli seconds

Application ServerDatabase

Server

Locally attached Disks


Client-Server Architecture with fast storage

Network roundtrip = 50 micro secApplication Server

Database Server

Latency dominated by network

SSD RAM

100 microsecs

100 nanosecs


Architecture of an Embedded Database


Application Server

Database Server




Application Server

Database Server

SSD RAM

100 microsecs

100 nanosecs




Application Server

Database Server

Storage attached directly to application servers

SSD RAM

100 microsecs

100 nanosecs


Any pre-existing embedded databases?

Open Source

FB Proprietary



1. Berkeley DB2. SQLite3. Kyoto TreeDB4. LevelDB

Open Source

FB Proprietary

Key-value stores



1. High Performant2. No transaction log3. Fixed size keys

Open Source

FB Proprietary

Key-value stores


Comparison of open source databases



Random Reads



Random Reads

Random Writes



Random Reads

LevelDBKyoto TreeDBSQLite3

129,000 ops/sec151,000 ops/sec134,000 ops/sec

Random Writes



Random Reads


129,000 ops/sec151,000 ops/sec134,000 ops/sec


164,000 ops/sec88,500 ops/sec

9,860 ops/sec

Random Writes


HBase and HDFS (in April 2012)

Details of this experiment: http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html


http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html



Random Reads






Random Reads

HDFS (1 node)HBase (1 node)

93,000 ops/sec35,000 ops/sec





Log Structured Merge Architecture

Read Write data in RAM




Write Request from Application





Transaction log





Transaction log Read Only data in RAM on

disk






disk

Periodic Compaction




Write Request from ApplicationScan Request from Application


disk

Periodic Compaction


Leveldb has low write ratesFacebook Application 1:

• Write rate 2 MB/sec only per machine

• Only one cpu was used


Leveldb has low write ratesFacebook Application 1:

• Write rate 2 MB/sec only per machine

• Only one cpu was used

We developed multithreaded compaction

10ximprovement on

write rate

100%of cpus are

in use+Monday, December 9, 13

Leveldb has stallsFacebook Feed:

• P99 latencies were tens of seconds

• Single-threaded compaction


Leveldb has stallsFacebook Feed:

• P99 latencies were tens of seconds

• Single-threaded compaction

We implemented thread aware compaction

Dedicated thread(s) to flush memtable Pipelined memtables

P99 reduced to less than a second


Leveldb has high write amplification•Facebook Application 2:

• Level Style Compaction

• Write amplification of 70 very high


Leveldb has high write amplification•Facebook Application 2:

• Level Style Compaction

• Write amplification of 70 very high

Two compactions by LevelDB Style Compaction

5 bytes

6 bytes

10 bytes

Level-0

Level-1

Level-2

Stage 1

11 bytes

10 bytes

Stage 2

10 bytes

Stage 3


Our solution: lower write amplification•Facebook Application 2:

• We implemented Universal Style Compaction

• Start from newest file, include next file in candidate set if

• Candidate set size >= size of next file


Our solution: lower write amplification•Facebook Application 2:

• We implemented Universal Style Compaction

• Start from newest file, include next file in candidate set if

• Candidate set size >= size of next file Single compaction by Universal Style Compaction

5bytes

6 bytes

10 bytes

Level-0

Level-1

Level-2

Stage 1

10 bytes

Stage 2

Write amplification reduced to <10Monday, December 9, 13

Leveldb has high read amplification



•Secondary Index Service:

•Leveldb does not use blooms for scans



•Secondary Index Service:

•Leveldb does not use blooms for scans

•We implemented prefix scans

• Range scans within same key prefix

• Blooms created for prefix

• Reduces read amplification


Leveldb: read modify write = 2X IOs



•Counter increments

• Get value, value++, Put value

• Leveldb uses 2X IOPS



•Counter increments

• Get value, value++, Put value

• Leveldb uses 2X IOPS

•We implemented MergeRecord

• Put “++” operation in MergeRecord

• Background compaction merges all MergeRecords

• Uses only 1X IOPS


Leveldb has a Rigid Design



•LevelDB Design

• Cannot tune system, fixed file sizes



•LevelDB Design

• Cannot tune system, fixed file sizes

•We wanted a pluggable architecture

• Pluggable compaction filter, e.g. TimeToLive

• Pluggable memtable/sstable for RAM/Flash

• Pluggable Compaction Algorithm


The Changes we did to LevelDB



Inherited from LevelDB

• Log Structured Merge DB

• Gets/Puts/Scans of keys

• Forward and Reverse Iteration



Inherited from LevelDB

• Log Structured Merge DB

• Gets/Puts/Scans of keys

• Forward and Reverse Iteration

RocksDB

• 10X higher write rate

• Fewer stalls

• 7x lower write amplification

• Blooms for range scans

• Ability to avoid read-modify-write

• Optimizations for flash or RAM

• And many more…


RocksDB is born!

•Key-Value persistent store

•Embedded

•Optimized for fast storage

•Server workloads


What is it not?

•Not distributed

•No failover

•Not highly-available, if machine dies you lose your data


RocksDB API▪ Keys and values are arbitrary byte arrays.

▪ Data is stored sorted by key.

▪ The basic operations are Put(key,value), Get(key), Delete(key) and Merge(key, delta)

▪ Forward and backward iteration is supported over the data.


RocksDB Architecture

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch



Write Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch



Write Request

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch



Write Request

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch

Memory

Switch



Write Request

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch

Memory Persistent Storage

Switch


Log Structured Merge Tree -- Writes

▪ Log Structured Merge Tree

▪ New Puts are written to memory and optionally to transaction log

▪ Also can specify log write sync option for each individual write

▪ We say RocksDB is optimized for writes, what does this mean?


RocksDB Write Path

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch


RocksDB Write Path

Write Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch


Log Structured Merge Tree -- Reads

▪ Data could be in memory or on disk

▪ Consult multiple files to find the latest instance of the key

▪ Use bloom filters to reduce IO


RocksDB Read Path

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Blooms


RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Blooms


RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Memory

Blooms


RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d


Blooms


RocksDB: Open & Pluggable

Pluggable Memtable

format in RAM

Blooms

CustomizableWAL



Pluggable Memtable

format in RAM


Blooms

CustomizableWAL



Pluggable Memtable

format in RAM


Transaction log

Blooms

CustomizableWAL



Pluggable Memtable

format in RAM


Transaction log Pluggable sst data format on storage

Blooms

CustomizableWAL



Pluggable Memtable

format in RAM



Pluggable Compaction

Blooms

CustomizableWAL



Pluggable Memtable

format in RAM

Write Request from ApplicationGet or Scan Request from Application


Pluggable Compaction

Blooms

CustomizableWAL


Example: Customizable WALogging

•In-house Replication solution wants to be able to embed arbitrary blob in the rocksdb WAL stream for log annotation

•Use Case: Indicate where a log record came from in multi-master replication

•Solution: A Put that only speaks to the log



Active MemTable

log

Replication Layer In one write batch:

PutLogData(“I came from Mars”)Put(k1,v1)

k1 v1 “I came from Mars”

k1/v1



Write Request

Active MemTable

log

Replication Layer In one write batch:

PutLogData(“I came from Mars”)Put(k1,v1)

k1 v1 “I came from Mars”

k1/v1


Example: Pluggable SST format

•One Facebook use case needs extreme fast response but could tolerate some loss of durability

•Quick hack: mount sst in tmpfs

•Still not performant:

• existing sst format is block based

•Solution: A much simpler format that just stores sorted key/value pairs sequentially

• no blocks, no caching, mmap the whole file

• build efficient lookup index on load


Example: Blooms for MemTable

•Same use case, after we optimized sst access, memtable lookup becomes a major cost in query

•Problem: Get needs to go through the memtable lookups that eventually return no data

•Solution: Just add a bloom filter to memtable!


RocksDB Read Path

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Blooms

Blooms

Blooms


RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Blooms

Blooms

Blooms


RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Memory

Blooms

Blooms

Blooms


RocksDB Read Path

Read Request

FlushCompaction

Active MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d


Blooms

Blooms

Blooms


Example: Pluggable memtable format

•Another Facebook use case has a distinct load phase where no query is issued.

•Problem: write throughput is limited by single writer thread

•Solution: A new memtable representation that does not keep keys sorted



Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch



Write Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch



Write Request

Read Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch Switch



Write Request

Read Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch

Memory

Switch



Write Request

Read Request

Sort, Flush

Compaction

Unsorted MemTable

ReadOnlyMemTable

log

log

log

LSM

sst sst

sst sst

sst

sst

d

Switch


Switch


Possible workloads for RocksDB?

▪ Serving data to users via a website

▪ A spam detection backend that needs fast access to data

▪ A graph search query that needs to scan a dataset in realtime

▪ Distributed Configuration Management Systems

▪ Fast serve of Hive Data

▪ A Queue that needs a high rate of inserts and deletes


Futures


Futures

•Scale linearly with number of cpus

• 32, 64 or higher core machines

• ARM processors


Futures

•Scale linearly with number of cpus

• 32, 64 or higher core machines

• ARM processors

•Scale linearly with storage iops

• Striped flash cards

• RAM & NVRAM storage


Come Hack with us


Come Hack with us

•RocksDB is Open Sourced

• http://rocksdb.org

• Developers group https://www.facebook.com/groups/rocksdb.dev/


http://rocksdb.org

http://rocksdb.org

https://www.facebook.com/groups/rocksdb.dev/


Come Hack with us

•RocksDB is Open Sourced

• http://rocksdb.org

• Developers group https://www.facebook.com/groups/rocksdb.dev/

•Help us HACK RocksDB


http://rocksdb.org

http://rocksdb.org



tech talk: rocksdb slides by dhruba borthakur & haobo xu of facebook

Technology

ramtransaction log

applicationtransaction

clientserver architecture

rates facebook application

facebook feed

random reads hdfs

random readsdetails

high level