the current messaging landscape: rabbitmq, zeromq, nsq, kafka
TRANSCRIPT
Messaging
We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases
A Global Mesh with a Memory
Millions of users
Dozens ofinternal clients
1988:(Harvard)
30,000
2020:(Global) 50,000,000,000
We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases
Messaging:
• What is it?• Why should I care?
Message
Event
Metric
Header
Body
Why should I care? Decoupling
by wmasterj and Fabrice TIERCELIN - Wikipedia
Why should I care? Decoupling
How does this happen?• Asynchronous interaction
A BMessages
Instances
Processes
Functions
AsyncScope
Time
P C
Idle
Idle
Idle
sync
async
Time
P C CP
Idle
Idle
Idle
Idle
sync vslatency
We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases
Broker
LAN
WAN
: + flexibility
LAN/WAN
DatabaseSemanticsFor access
:
+ scalability
GW GW …GW GW GW …GW
Region A Region B …
GW
Push Pullsnd rcv
µSvc …
: + IoT
2020:(Global) 50,000,000,000
:Written in C
+ IoT
GW
Push Pullsnd rcv
µSvc …∅mq
: + very fast∅mq
GW
Push Pull snd rcv
µSvc …NSQ
: + fastNSQ
GW
Push Pullsnd rcv
µSvc …NSQ
What if…
GW GW …GW GW GW …GW
Region A Region B …
Then:
… …
Region A Region B …
GW GW GW GW GW GW
Becomes:
: + offsets
We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases
: + flexibility
: + very fast∅mq
: + fastNSQ
: + IoT
: + offsets
Distributed Graphs at Scale with Cassandra and Titan
● Structured Knowledge Representation
● Index-free adjacency: Like memory pointers, but in disk
● Navigation between nodes in constant time.
● Graph != No schema
Michael LaingArchitect, Edge Engineering (soon to include OLAP w Spark Streaming)
New York [email protected]