2015-08-22 stream billions of data to google big query
TRANSCRIPT
![Page 2: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/2.jpg)
Shipment tracking APISupporting 270 290+ carriers
![Page 3: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/3.jpg)
Free Shipping APIsSingle API for 5 carriers
![Page 4: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/4.jpg)
How do we find you?
![Page 5: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/5.jpg)
Google BigQuery is great
- Charge only when you do query
- Super cheap in storage
- Super cheap in sending the data
![Page 6: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/6.jpg)
Perfect for log everything
3 billions records / month
1,200 records / sec
![Page 7: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/7.jpg)
API Log BigQuery
Goal
![Page 8: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/8.jpg)
API Log BigQuery
It does not work!
- http RESTful- BigQuery API call limit
![Page 9: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/9.jpg)
API Log Redis Streamer BigQuery
How do we solve?
![Page 10: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/10.jpg)
Why Redis?
- TCP persistent connection- Fast, really FAST
![Page 11: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/11.jpg)
What does the STREAMER do?
- Read data from redis- Send BATCH data to BQ- Pre / Post process data
![Page 12: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/12.jpg)
Demo
![Page 13: 2015-08-22 Stream billions of data to Google Big Query](https://reader035.vdocuments.us/reader035/viewer/2022062820/58aad7f21a28ab27178b4d61/html5/thumbnails/13.jpg)
http://www.slideshare.net/aftership
FAQ
OPEN SOURCEhttps://github.com/AfterShip/demo_streamer2bigquery