hbase/phoenix @ scale
TRANSCRIPT
![Page 1: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/1.jpg)
HBase/PH ENIX @ ScaleA study of Salesforce’s use of HBase and Phoenix
Lars Hofhansl
Vice President and Principal Architect at Salesforce
Apache HBase, Apache Phoenix committer and PMC member
![Page 2: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/2.jpg)
Two Years Ago, I showed you this
![Page 3: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/3.jpg)
![Page 4: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/4.jpg)
Zookeeper?
![Page 5: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/5.jpg)
Zookeeper?
HBase?
![Page 6: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/6.jpg)
Zookeeper?
HBase?
HDFS?
![Page 7: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/7.jpg)
Zookeeper?
Commodity
Hardware?
HBase?
HDFS?
![Page 8: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/8.jpg)
Zookeeper?
Commodity
Hardware?
HBase?
HDFS?Unstructured
Data?
![Page 9: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/9.jpg)
HBase/Phoenix are BIG* at Salesforce
* Numbers are from some time in the past and do not reflect the current scale
![Page 10: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/10.jpg)
Heavy users of relational and semi-structure data
![Page 11: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/11.jpg)
Mix of customer* and internal data
* Through the Salesforce Platform Offering
![Page 12: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/12.jpg)
Typical Use Cases
Samples of Customer Data:
Login data to track anomalies in real-time
Archiving, historical data moved from operational, relational storage to HBase
Denormalized feed views for Salesforce Chatter
Chatter @mention low-latency relevancy queries
Storage of user activity on marketing campaigns for reporting and AI/ML
Samples of Internal usage:
Periodic thread dumps from all AppServers
Machine metrics from all machines
![Page 13: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/13.jpg)
Customer Facing
![Page 14: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/14.jpg)
> 100 clusters of varying size
![Page 15: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/15.jpg)
~4bn write requests / day
![Page 16: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/16.jpg)
~80TB written / day
![Page 17: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/17.jpg)
That’s about ~8 gbit/s, sustained
![Page 18: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/18.jpg)
~600m read requests / day
![Page 19: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/19.jpg)
~500GB read / day
![Page 20: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/20.jpg)
Central Metrics Database
![Page 21: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/21.jpg)
Central Metrics Database
![Page 22: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/22.jpg)
Collecting data from > 80.000 machines
![Page 23: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/23.jpg)
11.4 trillion metrics stored and growing
![Page 24: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/24.jpg)
2.8 tn metrics in 6 months and growing
![Page 25: HBase/PHOENIX @ Scale](https://reader030.vdocuments.us/reader030/viewer/2022021503/5a647a867f8b9a63568b472b/html5/thumbnails/25.jpg)
210 bn reads in 6 months