firehose
DESCRIPTION
how we built a firehose with NodeJS and ZeroMQTRANSCRIPT
![Page 1: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/1.jpg)
FirehoseBuilding a real-time data stream with NodeJS
Luca Grulla@ForwardLondon, 2/8/2011
![Page 2: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/2.jpg)
Introduction
Or what we wanted to build
![Page 3: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/3.jpg)
We like to have a (near) real-time view of the data stream
![Page 4: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/4.jpg)
Resilient
![Page 5: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/5.jpg)
Scalable
![Page 6: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/6.jpg)
Accurate(but not necessarily precise)
![Page 7: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/7.jpg)
![Page 8: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/8.jpg)
Verse and Chorus
Or how we built it
![Page 9: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/9.jpg)
Stack
• NodeJS (0.4.9)• CoffeeScript• ZeroMQ
![Page 10: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/10.jpg)
Architecture
![Page 11: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/11.jpg)
Interlude
Or joys and pitfalls of building a tail library for node
![Page 12: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/12.jpg)
Living the asynchronous dream
![Page 13: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/13.jpg)
Living the asynchronous dream
• Error manifests itself only at high throughput (7MB/secs)
• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)
![Page 14: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/14.jpg)
Breathing in an asynchronous way
![Page 15: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/15.jpg)
Learnings
• You need to think asynchronous to write asynchronous code
• Dispatch internal events if you need to keep control of the flow
• “Internal queues” to scale(up to a certain limit)
![Page 16: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/16.jpg)
Performance
• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput
![Page 17: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/17.jpg)
Questions ?
![Page 18: Firehose](https://reader035.vdocuments.us/reader035/viewer/2022062307/55493fc3b4c9050f4d8b4ffd/html5/thumbnails/18.jpg)
References
• npm install tail• https://github.com/forward/node-tail