tailored database solutions: building a 100 billion records db in 20 minutes
DESCRIPTION
In this talk, we can see how we can use Voron to build a time series database to store 100 billion recordsTRANSCRIPT
Tailored database solutions with Voron
Oren [email protected] ayende.com/blog
Hibernating RhinosLive
Cod
ing
Sess
ion
Let’s implement a database Time series database, but what are those?
Heart rate monitor watch. Every 30 second sends:
Watch Id, Time, Heart Rate
You sold 100,000 of those. Users want to log in and see their stats.
Traffic Per watch:
2,880events per day
89,280events per month
1,051,200events per year
Globally (100K watches): 288,000,000
events per day 8,928,000,000
events per month 105,120,000,000
events per year
Just under 3,500 events per second, sustained
100 billion events / year How do we approach this?
Design… Each watch has an id (string, up to 40 chars) We model each watch as a tree in Voron The tree uses:
Big Endian date for key Heart rate number as value
Let us go and build this…
Let us build this
And someone time me…, please
Great, now about those reads What kind of reads we need? Always on a specific watch. For a given date range.
Let us build this
And someone time me…, please
What is left? Real performance testing… Concurrent reads & writes Connecting to a server…
Dropping old data / consolidating it.
Questions?