big data messaging
TRANSCRIPT
![Page 1: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/1.jpg)
Messaging
![Page 2: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/2.jpg)
○ Messaging models○ Protocols & Standards○ Message brokers○ Data pipeline architecture○ Microservices architecture
Agenda
![Page 3: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/3.jpg)
Overview
![Page 4: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/4.jpg)
Messaging models
Publisher-Subscriber
Point-to-Point
![Page 5: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/5.jpg)
○ AMQP - Advanced Message Queuing Protocol○ MQTT - Message Queuing Telemetry Transport
(machine-to-machine (M2M)/"Internet of Things" connectivity protocol)
○ STOMP - Simple (or Streaming) Text Orientated Messaging Protocol
○ JMS - Java Message Service
Messaging protocols & standards
![Page 6: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/6.jpg)
RabbitMQ○ You have messages that need
to be routed in complex ways to consumers
○ You want per-message delivery guarantees
○ You don't care about ordered delivery
Message broker examplesApache Kafka
○ You have a fire hose of events
○ You need delivered in partitioned order 'at least once'
○ You want to be able to re-read messages
https://www.quora.com/What-are-the-differences-between-Apache-Kafka-and-RabbitMQhttp://queues.io
![Page 7: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/7.jpg)
Pipeline is a set of data processing elements connected into series, where the output of one element is the input of the next one.
Data pipeline architecture
![Page 8: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/8.jpg)
Microservices architecture
![Page 9: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/9.jpg)
Apache Kafka Hits 1.1 Trillion Messages Per Day – Joins the 4 Comma Club
Apache Kafka at LinkedIn
![Page 10: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/10.jpg)
Apache Kafka technical overview
![Page 11: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/11.jpg)
Apache Kafka technical overview
![Page 12: Big data messaging](https://reader035.vdocuments.us/reader035/viewer/2022070603/587181191a28ab230b8b78f1/html5/thumbnails/12.jpg)
Questions?