the hive think tank: rocking the database world with rocksdb
TRANSCRIPT
![Page 1: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/1.jpg)
![Page 2: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/2.jpg)
Caching a Billion Items: A Case Study of Rakuten Programmatic DSP
Qian ZhuPrinciple EngineerRakuten Marketing
![Page 3: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/3.jpg)
About Me• Principle engineer, Rakuten• Worked in Accenture Technology Labs for 4 years
– Infrastructure and System software• Research focus on big data infrastructure and analytics• Ph.D. in Computer Science and Engineering,
specializing on distributed systems and data mining
![Page 4: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/4.jpg)
About Rakuten• 3rd largest e-commerce retailer in the world• Largest consumer services company in Japan• Rakuten Affiliate Network (formerly Rakuten LinkShare)
is a leading provider of full-service online marketing solutions – voted #1 for 4 years in a row
![Page 5: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/5.jpg)
![Page 6: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/6.jpg)
Retargeting Platform
• The platform delivers real-time segment loading, scalability, frequency capping, AdX inventory and AdX automatic creative tagging
• the architecture lays the foundation for extensive data flow validations, business alerts and performance optimization
![Page 7: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/7.jpg)
Challenges• Low latency: respond to bid requests in 10s of ms
– 100 ms on bidding– generating retargeting data within a few seconds
• High I/O: handle intensive (random) query workload– Up to 100,000 QPS per bidder– Tens of bidders across the world
• Parallelism: write, read and data lifetime management in parallel
• Easy migration, eventual consistency and etc.
![Page 8: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/8.jpg)
![Page 9: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/9.jpg)
How RocksDB is Used• Write thread
– Multiple column families for segment data, segment index*, Kafka offsets and expire index
• Query thread– Range scan and random single record retrieval
• Expiry thread– Triggered at a demanded frequency– Delete data from both RocksDB and bidder cache
![Page 10: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/10.jpg)
Moving Forward• Apply for other Rakuten services
• We are hiring!!!– System programmers– Data engineers– Machine learning experts
![Page 11: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/11.jpg)
Questions?• https://www.linkedin.com/in/qian-zhu-40a3917• mailto: [email protected]
![Page 12: The Hive Think Tank: Rocking the Database World with RocksDB](https://reader036.vdocuments.us/reader036/viewer/2022062311/587146881a28ab55588b5929/html5/thumbnails/12.jpg)