High Concurrency & Throughput Using MQTT and Kafka

by Evan Purnama (Co-Founder, CTO Qiscus Pte Ltd)

Introduction

What is Qiscus?

We are a startup company that help other companies to enable conversations.

What is Qiscus?

What is Qiscus?

What is Qiscus?

Future of Technology

Future of Technology

Future of Technology

All Of Those Translate to ...

Huge Concurrent Connections

Huge Number of Messages Being

Processed

MQTT (Message Queuing Telemetry Transport)

- Created at 1999 by IBM.- Optimised for constrained environment.- Being used for communicate in rural/offshore area.

Pub Sub

MQTT

- Broker and Client (Publisher & Subscriber). - Highly Scalable Architecture- Broker become Single Point of Failure (SPoF).

- Both publishers and subscribers are very simple- Lightweight ( ~ few bytes overhead per message)- Very little resources on client side

- MQTT Broker is complex while the client is thin- Suitable for constrained environment

MQTT

- Reliable- QoS 0, 1, 2- Last Will & Testament Messages

- Secure- Username & Password- TLS

Components of MQTT

- Publisher and Subscriber- Topics- Message- QoS- Broker- etc.

Topics

- “Address” for client side to Publish or Subscribe.- Patterns :

- foo/abc/bar- foo/+/bar- foo/#

- Creating topic is cheap.- Use topic for specific event instead of general event.

Broker

- Open Source Based- Mosquitto- EMQ- Mosca- VerneMQ

- Managed Service MQTT Broker

Scaling Goal

Scaling Goals

- Connect as many connected users/devices as possible.- Handle as many throughput as possible

Scaling Goals

- Cloud based (AWS, GCP, etc)- Horizontal Scaling

- reduce cost- plan for future- less impact downtime

Concurrent Connections

Publisher Publisher Publisher Subscriber Subscriber Subscriber

Single MQTTBroker

Scaling Concurrent Connections

Publisher Publisher Publisher Subscriber Subscriber Subscriber

MQTTBroker 2

MQTTBroker 1

Scaling Concurrent Connections

Publisher Publisher Publisher Subscriber Subscriber Subscriber

MQTTBroker 2

MQTTBroker 1

Shared Memory Data

Scaling Concurrent Connections

Publisher Publisher Publisher Subscriber Subscriber Subscriber

MQTTBroker 2

MQTTBroker 1

Scaling Concurrent Connections

Load Balancing

- Which broker to connect to?- DNS Load Balancing.

High Concurrent Connections

Publisher Publisher Publisher Subscriber Subscriber Subscriber

MQTTBroker 2

MQTTBroker 1

DNS Load Balancer

Message Processing

Message Processing

Message Processing

Publisher

Publisher

Publisher

Publisher

Publisher

ClusterBroker MQTT

Subscriber

Subscriber

DB

Elastic Search

Push Notif, etc

Message Processing

- Sending to multiple destinations from the same instance that receiving very huge throughput is not a scalable approach.

- Adding more component means need to add to all of the subscribers.- What if one of the component suddenly become unavailable?

- Data lost?- Subscriber need to replay?

Kafka

- Unified, high-throughput, low-latency platform for handling real-time data feeds.

- Highly scalable stream, with throughput up to ~200k-300k for single node.

- Originally from LinkedIn, to solve Log Aggregation problems.- Zoo-Keeper- Publisher, Consumer, Topic, Partition concept.- Very scalable server-to-server messaging system.

Kafka

Scalable Message Processing

Publisher

Publisher

Publisher

Publisher

Publisher

ClusterBroker MQTT

DB

Elastic Search

Push Notif, etc

Subscriber

Subscriber

ClusterKafka

Notes :

- This strategy is only suitable for burst- need to make sure that the “consumer” able to catch-up to process

the input stream.- Kafka becoming Single Point of Failure (SPoF) on message processing,

be careful with this. Make sure to setup properly and make it High Availability (HA).

Thank You

References :

- https://blog.qiscus.com/handle-10-million-concurrent-user-connections-5-millions-messages-per-minute-throughput/

- https://www.youtube.com/watch?v=WFfciQ1U_lU

- https://www.csoonline.com/article/3346082/what-is-shadow-iot-how-to-mitigate-the-risk.html

- https://www.fastcompany.com/4011195/this-ibm-smart-bus-comes-with-a-watson-powered-assistant

- https://youngster.id/featured/vendor-telekomunikasi-berperan-penting-dalam-melindungi-smart-home/