building a database from scratch - erlang-factory.com · versatile data endpoint micro-services,...
TRANSCRIPT
![Page 1: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/1.jpg)
EUC 2017 Stockholm - 06/2017
Building a database from scratch
![Page 2: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/2.jpg)
benoît chesneau
craftsman working on P2P and custom data endpoints technologies
opensource only
enki multimedia : the corporate interface
about me
![Page 3: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/3.jpg)
versatile data endpoint
micro-services, message solutions are all based about custom data endpoints
need for a simple solution that allows you to bring the data near your service or locally.
why barrel?
![Page 4: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/4.jpg)
a modern database
documents, with time and attachments
distributed, local first
bring a view of your data near your application
automatic indexing
focus on simplicity
what is barrel?
![Page 5: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/5.jpg)
distributed: P2P
![Page 6: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/6.jpg)
query
a partial view of the data
node
node
![Page 7: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/7.jpg)
local databasemobile
sensor
"cloud" database
local database
a partial view of the data
![Page 8: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/8.jpg)
agnostic indexing
![Page 9: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/9.jpg)
barrel can be embedded in your own Erlang application:
local database
no need to cache
platform release: HTTP/Erlang pod to store and query the documents
platform
![Page 10: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/10.jpg)
problems to solve
![Page 11: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/11.jpg)
stateful
different queries return different results
update expectations
read your own write?
database complexity
![Page 12: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/12.jpg)
processes don’t share anything
how do we have multiple writers and multiple readers
actor model
no integer atomic operations
IO operations are “slow”
until you get nifs
erlang constraints
![Page 13: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/13.jpg)
build over existing storage solutions:
key/value interface
allows atomic batch updates
ordered set
1 collection, 1 storage
collections are small
decisions
![Page 14: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/14.jpg)
hierachical
dbs db docs
a collectionmultiple collections on a node
store
![Page 15: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/15.jpg)
document:
map in erlang
revision tree:
https://oceanstore.cs.berkeley.edu/publications/papers/pdf/hh_icdcs03_kang.pdf
storing a document
![Page 16: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/16.jpg)
revision tree
![Page 17: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/17.jpg)
2 modes: lazy and consistent
lazy: indexed asynchronously based on the changes feed
consistent
support maps, filter, chain opererations based on paths
indexing
![Page 18: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/18.jpg)
internals
![Page 19: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/19.jpg)
using rocksdb for the storage
http://gitlab.com/barrel-db/erlang-rocksdb
used for memory and disk. optimised for SSD.
dirty nifs
rocksdb
![Page 20: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/20.jpg)
barrel_sup db_sup db
db
db supervision
store
![Page 21: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/21.jpg)
writes are queued on the main db process
store a canonical version of doc
states of the database is shared between other processes via ETS
readers are getting last db state via ets
write process (current)
![Page 22: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/22.jpg)
prevent delayed jobs
![Page 23: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/23.jpg)
write more operations at once
selective receive
group operations based on the document ID (merge)
from 40 RPS to 1000 RPS on a node with 4GB of ram and 2 cores)
write process (current)
![Page 24: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/24.jpg)
By ID, Changes queries
get latest DB state from ETS
everything happen on the reader process
coming: backpressure
share the db state across a pool of readers
remove the state from ETS
readers
![Page 25: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/25.jpg)
testing dispatching of write operations on different processes:
https://arxiv.org/pdf/1509.07815.pdf
testing optimistic writes
back pressure:
short circuit to not accept more write than the node can sustain
based on the running transaction and metrics
similar to safety valve: https://github.com/jlouis/safetyvalve
write process rewrite
![Page 26: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/26.jpg)
just appending data to the storage we never read from old index values
inside the DB process for consistent write
a process listening on db updates events (using a simple gen_server, no gen_event)
index policies to index each json segment to retrive via their valur or hash to support value or range queries.
indexing process
![Page 27: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/27.jpg)
over HTTP
cowboy 2
over TCP using teleport and Erlang serialisation (coming):
https://gitlab.com/barrel-db/teleport
allows embedded mode
replication
![Page 28: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/28.jpg)
add some instrumentation
![Page 29: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/29.jpg)
how to not block without counting
first try: statsd client sending to an UDP endpoint counter/gauge/histogram updates
we run out of processes & file descriptors
asynchronous sending: better.
how to make generic?
instrumenting
![Page 30: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/30.jpg)
add hooks
https://github.com/benoitc/hooks
prometheus plugin and wombat support (EE version)
internal metrics sytem
https://gitlab.com/barrel-db/lab/instrument
instrumenting
![Page 31: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/31.jpg)
roamap
![Page 32: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/32.jpg)
0.9 release: 2017/06/13
https://gitlab.com/barrel-db/barrel-platform
add documentation (june 2017)
optimise writing
atomic updates
enrich query engine.
roadmap
![Page 33: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/33.jpg)
?
![Page 34: Building a database from scratch - erlang-factory.com · versatile data endpoint micro-services, message solutions are all based about custom data endpoints need for a simple solution](https://reader034.vdocuments.us/reader034/viewer/2022050208/5f5ae0581fdad5414607fb41/html5/thumbnails/34.jpg)
twitter: @barreldb web: https://barrel-db.org
contact