scaling mqtt with apache kafka

29
SCALING MQTT WITH KAFKA Tim Kellogg April 7, 2014 @kellogh

Post on 17-Oct-2014

2.487 views

Category:

Technology


3 download

DESCRIPTION

My slides for ApacheCon North America 2014.

TRANSCRIPT

Page 1: Scaling MQTT With Apache Kafka

SCALING MQTT WITH KAFKA Tim Kellogg April 7, 2014 @kellogh

Page 2: Scaling MQTT With Apache Kafka

•  MQTT broker •  Protocol onboarding •  Cloud environment (we’re a startup)

Page 3: Scaling MQTT With Apache Kafka
Page 4: Scaling MQTT With Apache Kafka

•  Standard •  Lightweight

o  >= 2 byte overhead per message

•  Easy to parse o  Length prefixed strings

•  Requires very little resources on client side o  Broker keeps track of state

Page 5: Scaling MQTT With Apache Kafka

•  Reliable o  QoS 1 & 2 o  Last Will & Testament messages

•  Secure o  Username + Password o  Tunnel over TLS

Page 6: Scaling MQTT With Apache Kafka

Publish / Subscribe Pub

Pub

Pub

Broker

Sub

Topic/A

Topic/B

Topic/C

Topic/B

Sub Topic/C

Sub Topic/A

Page 7: Scaling MQTT With Apache Kafka

Topics •  foo/bar/baz •  com.example/device/17/thermo • Patterns

•  com.example/device/+/thermo •  com.example/device/#

Page 8: Scaling MQTT With Apache Kafka

Scaling Goals

•  More than 2 Million connected publishers

•  More than 65,000 msg/s

•  Single subscriber

Page 9: Scaling MQTT With Apache Kafka

Scaling Goals •  Amazon’s EC2 •  Horizontal scaling

o  Reduce cost o  Plan for the future o  Less impact from

downtime

Page 10: Scaling MQTT With Apache Kafka

Problems with Scaling MQTT

Page 11: Scaling MQTT With Apache Kafka

Load Balancing •  Which broker to connect to?

o  DNS load balancing

•  HAProxy •  QoS 1-2 messages stored in

Cassandra o  Consistent hash ring

Page 12: Scaling MQTT With Apache Kafka

Single Subscriber Pub

Pub

Pub

Broker

Sub

Topic/A

Topic/B

Topic/C

Topic/#

Page 13: Scaling MQTT With Apache Kafka

Single Subscriber Pub

Pub

Pub

Broker Sub

Topic/A

Topic/B

Topic/C

Topic/#

Broker

Broker

Load Balancing

Page 14: Scaling MQTT With Apache Kafka

Single Subscriber Pub

Pub

Pub

Broker Sub

Topic/A

Topic/B

Topic/C

Topic/#

Broker

Broker

Load Balancing

Load Balancing

Page 15: Scaling MQTT With Apache Kafka

Single Subscriber

Broker

Subscriber

Topic/#

Broker

Broker

Page 16: Scaling MQTT With Apache Kafka

Using HTTP

Page 17: Scaling MQTT With Apache Kafka

POST From The Broker Pub

Pub

Pub

Broker

Topic/A

Topic/B

Topic/C Broker

Broker HTTP POST

Load Balancing

Server

HTTP POST Server

HTTP POST Server

Load Balancing

Page 18: Scaling MQTT With Apache Kafka

Benefits • Easy to load balance • Well known & well

supported

Page 19: Scaling MQTT With Apache Kafka

Drawbacks • HTTP is heavy

•  Headers •  Creating & destroying TCP connections

• Subscriber servers must be available

•  Retry logic to guarantee delivery

Page 20: Scaling MQTT With Apache Kafka

Apache Kafka

Page 21: Scaling MQTT With Apache Kafka

•  •  Distributed log

aggregation framework •  Server to server •  “Smart” clients •  Apache ZooKeeper

Page 22: Scaling MQTT With Apache Kafka

•  Append-only files per topic o  Client keeps track of what messages it’s processed

•  No topic wildcards •  Key is used for out of band data •  device/42/thermo è topic: device-thermo key: 42

Page 23: Scaling MQTT With Apache Kafka

Subscriber Group Pub

Pub

Pub

Broker

Page 24: Scaling MQTT With Apache Kafka

Subscriber Group Pub

Pub

Pub

Broker

Broker

Broker

Load Balancing

Kafka

Page 25: Scaling MQTT With Apache Kafka

Results

Page 26: Scaling MQTT With Apache Kafka

•  Linear scaling for fire hose subscriber •  At least 2 million clients •  At least 65,000 msg/s

Page 27: Scaling MQTT With Apache Kafka

Wish List •  Security •  Configuration

Page 28: Scaling MQTT With Apache Kafka

Open Source IoT

Page 29: Scaling MQTT With Apache Kafka

The Book: Mastering The Internet of Things

Questions?

@kellogh