journey to the real-time analytics in extreme growth
TRANSCRIPT
Real Time Dashboard
• User acquisition• 8B events daily
Data is Mutable
Previous solution - Toku (Mongo)
K
AF
KA
Toku writers Toku master
Toku slavesDashboard
Toku Problems
• Failures on weekly basis• Bad modeling • No recovery
Requirements
• RealTime• More events (more data)• More dimensions (MUCH MORE DATA !!!)• Stability• Faster
Dashboard - DB abstraction level
KA
FK
A
Toku writers Toku master
Toku slavesDashboard Middleware
(Vishnu)
We tried...
https://www.meetup.com/Druid-Israel/events/232075974/
What did we gain?
• Flexible middleware• Batch daily process - first step to recovery • Developers Paradise
Down to Earth
MemSQL
In Memory Scalable DB
Current Solution - MemSQL
MemSQL ArchitectureK
AF
KA
MemSQL writers Memsql
Cluster
Dashboard
Middleware(Vishnu)
MemSQL writers
MemsqlCluster (Slave)
Recovery K
AF
KA
(24
h) MemSQL
writersMaster
MemsqlCluster
Dashboard
Middleware(Vishnu)
Yesterday snapshot
RecoveryMemsqlCluster
MemSQL writers - only current day
Mem SQL - Quick Win
• Fast• Recoverable• Possibility to return to 0 point• Ability to add new features• More Data (X30)
Show me the numbers
• Data - 100 GB x 2 clusters• Query Latency - 1-3 seconds• Machines x 2 clusters
– 2 aggregators - m4.4xlarge– 4 leaves - r3.4xlarge
• Cost reduction $20K less than toku monthly
Good Enough Approach
• More data - more money• Less money - less data
Current - ArchitectureK
AF
KA
writers - only new
data
MemsqlRowstore
Cluster1-2 weeks
Dashboard
Middleware(Vishnu)Daily Batch
process
S3 files
MemsqlColumnstore
HistoryCluster
Daily
“Premature optimization is a root of all evil”
Donald Knuth
appsflyer.com/jobs
http://www.shutterstock.com/pic.mhtml?utm_campaign=ClipartLogo&irgwc=1&tpl=46764-50655&id=154723511&language=en&utm_medi
um=Affiliate&utm_source=46764
http://www.samatters.com/wp-content/uploads/2015/07/round-peg.jpg
http://marsmedia.info/en/blog/cassandra.png
http://www.zdnet.de/wp-content/uploads/2013/10/mongodb-logo.jpg
https://chris.lu/upload/images/redis.png
https://upload.wikimedia.org/wikipedia/en/b/ba/Druid_MasterLogo_Full_Color_Small.png
https://www.leftronic.com/wp-content/uploads/2015/04/Amazonredshift_220x110.png