firehose
Post on 06-May-2015
2.790 Views
Preview:
DESCRIPTION
TRANSCRIPT
FirehoseBuilding a real-time data stream with NodeJS
Luca Grulla@ForwardLondon, 2/8/2011
Introduction
Or what we wanted to build
We like to have a (near) real-time view of the data stream
Resilient
Scalable
Accurate(but not necessarily precise)
Verse and Chorus
Or how we built it
Stack
• NodeJS (0.4.9)• CoffeeScript• ZeroMQ
Architecture
Interlude
Or joys and pitfalls of building a tail library for node
Living the asynchronous dream
Living the asynchronous dream
• Error manifests itself only at high throughput (7MB/secs)
• The few implementations you can find around are wrong (including Log.io – we are planning to contribute back)
Breathing in an asynchronous way
Learnings
• You need to think asynchronous to write asynchronous code
• Dispatch internal events if you need to keep control of the flow
• “Internal queues” to scale(up to a certain limit)
Performance
• Beyond 5MB we start to fill the internal queue• We are limited by IO, not from CPU/memory• Consistent throughput
Questions ?
References
• npm install tail• https://github.com/forward/node-tail
top related