scaling an api: from reboots to redis, how (not) to do it
DESCRIPTION
How do you massively scale an API as your business grows? Ciaran worked for a company that has grown from a small B2C app written in PHP/MySQL to a B2B advertising company doing 15 Billion API calls per month. Along the way there have been some big technology changes and decisions that have both helped and hindered us. Along the way we have had 3 day outages, several AWS outages, massive traffic spikes especially early days when we signed customers like Wordpress.com, and many many improvements, re-factors and even successes. Today and for the last year we are highly modern engineering team, using many of the latest technologies and methodologies. We code in PHP and Python with C, Go, CoffeeScript and more, store data in MongoDB, Redis, memcache, Solr and MySQL, we have a full CI environment, do deploys using IRC (and Hubot) and Puppet everything. We have 100% (not 99.999%) up time for over 12 months now.TRANSCRIPT
![Page 1: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/1.jpg)
Scaling an APIFrom Reboots to Redis, how (not) to do it
![Page 2: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/2.jpg)
{ "fullName": "Ciarán Rooney", "work": { "position": ["CTO", "Speaker", "Mentor"] }, "home": { "email": "[email protected]", "twitter": "@CiaranR", "website": "http://ciaran.ie", "hobby": "http://phplondon.org" }}
![Page 3: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/3.jpg)
![Page 4: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/4.jpg)
{ "dataYear": "2013", "dataMonth": "april", "dataPeriod": "30 days", "dataRequests": { "apiRequests": "18,256,654,541", "impressions": “5,773,142,857", "uniqueUsers": "145,137,679" }, "customers": { "websitesPowered": "1,087,348", "customers": "26,379" }}
![Page 5: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/5.jpg)
![Page 6: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/6.jpg)
![Page 7: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/7.jpg)
![Page 8: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/8.jpg)
![Page 9: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/9.jpg)
![Page 10: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/10.jpg)
![Page 11: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/11.jpg)
![Page 12: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/12.jpg)
![Page 13: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/13.jpg)
![Page 14: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/14.jpg)
![Page 15: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/15.jpg)
BIND / DNS
imageStorage
![Page 16: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/16.jpg)
imageStorage
![Page 17: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/17.jpg)
imageStorage
![Page 18: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/18.jpg)
![Page 19: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/19.jpg)
![Page 20: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/20.jpg)
![Page 21: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/21.jpg)
![Page 22: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/22.jpg)
![Page 23: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/23.jpg)
APIs
![Page 24: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/24.jpg)
Database Master/Slave
![Page 25: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/25.jpg)
Client App
![Page 26: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/26.jpg)
Client App
Database Master/Slave
APIs
![Page 27: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/27.jpg)
![Page 28: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/28.jpg)
WordPress
![Page 29: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/29.jpg)
WordPress Pinterest
![Page 30: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/30.jpg)
Client App
Database Master/Slave
APIs
![Page 31: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/31.jpg)
WordPress
![Page 32: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/32.jpg)
![Page 33: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/33.jpg)
![Page 34: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/34.jpg)
SkimWords™
![Page 35: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/35.jpg)
![Page 36: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/36.jpg)
![Page 37: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/37.jpg)
APIxx
![Page 38: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/38.jpg)
APIxx APIxx APIxx APIxx APIxx APIxx APIxx APIxx
CLUSTERxx
ELBELB
![Page 39: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/39.jpg)
CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxxCLUSTERxx
US-WEST US-EAST EU-WEST
![Page 40: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/40.jpg)
![Page 41: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/41.jpg)
![Page 42: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/42.jpg)
![Page 43: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/43.jpg)
![Page 44: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/44.jpg)
![Page 45: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/45.jpg)
![Page 46: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/46.jpg)
![Page 47: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/47.jpg)
JAYxx
Apache Flume
![Page 48: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/48.jpg)
JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx JAYxx
ELBELB
CLUSTERxx
Apache FlumeCollector
C Prefix Tree C NLP Filterin
g
![Page 49: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/49.jpg)
CLUSTERxx CLUSTERxx CLUSTERxx CLUSTERxxCLUSTERxx
US-WEST
US-EAST EU-WEST
Apache FlumeMaster
![Page 50: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/50.jpg)
![Page 51: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/51.jpg)
Cacti
![Page 52: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/52.jpg)
![Page 53: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/53.jpg)
Conclusions
Small Scalable units
Horizontal is always better
Eliminate Single points of failure
Scale writes and reads separately
Automate everything
Monitor everything
![Page 54: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/54.jpg)
Scaling an APIFrom Reboots to Redis, how (not) to do it
Questions?
![Page 55: Scaling an API: From Reboots to Redis, how (not) to do it](https://reader033.vdocuments.us/reader033/viewer/2022052903/5575e417d8b42af74e8b4832/html5/thumbnails/55.jpg)
{ "fullName": "Ciarán Rooney", "workLife": { "position": "CTO" }, "homeLife": { "email": "[email protected]", "twitter": "@CiaranR", "website": "http://ciaran.ie", "hobby": "http://phplondon.org" }}