the current messaging landscape: rabbitmq, zeromq, nsq, kafka

Messaging

Michael LaingArchitect, Edge Engineering

New York Timesmichael.laing@nytimes.com

We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases

A Global Mesh with a Memory

Millions of users

Dozens ofinternal clients

1988:(Harvard)

30,000

2020:(Global) 50,000,000,000

We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases

Messaging:

• What is it?• Why should I care?

Message

Event

Metric

Header

Body

Why should I care? Decoupling

by wmasterj and Fabrice TIERCELIN - Wikipedia

Why should I care? Decoupling

How does this happen?• Asynchronous interaction

A BMessages

Instances

Processes

Functions

AsyncScope

Time

P C

Idle

Idle

Idle

sync

async

Time

P C CP

Idle

Idle

Idle

Idle

sync vslatency

We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases

Broker

LAN

WAN

: + flexibility

LAN/WAN

DatabaseSemanticsFor access

:

+ scalability

GW GW …GW GW GW …GW

Region A Region B …

GW

Push Pullsnd rcv

µSvc …

: + IoT

2020:(Global) 50,000,000,000

:Written in C

+ IoT

GW

Push Pullsnd rcv

µSvc …∅mq

: + very fast∅mq

GW

Push Pull snd rcv

µSvc …NSQ

: + fastNSQ

GW

Push Pullsnd rcv

µSvc …NSQ

What if…

GW GW …GW GW GW …GW

Region A Region B …

Then:

… …

Region A Region B …

GW GW GW GW GW GW

Becomes:

: + offsets

We’ll talk about:• nyt aбrik (my project)⨍• Messaging• Messaging OSS• Use cases

: + flexibility

: + very fast∅mq

: + fastNSQ

: + IoT

: + offsets

Distributed Graphs at Scale with Cassandra and Titan

● Structured Knowledge Representation

● Index-free adjacency: Like memory pointers, but in disk

● Navigation between nodes in constant time.

● Graph != No schema

Michael LaingArchitect, Edge Engineering (soon to include OLAP w Spark Streaming)

New York Timesmichael.laing@nytimes.com