intro to redis streams · who we are 3 open source. the leading in-memory database platform,...
TRANSCRIPT
![Page 1: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/1.jpg)
Intro to Redis StreamsIMCSUMMIT - NOVEMBER 2019 | DAVE NIELSEN
![Page 2: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/2.jpg)
2
• What is a data stream?
• What are the typical challenges you face while managing data streams?
• What is Redis Stream?
• How Redis Stream addresses the challenges?
• How to use Redis Streams?
![Page 3: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/3.jpg)
Who We Are
3
Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case.
The open source home and commercial provider of RedisEnterprise technology, platform, products & services.
![Page 4: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/4.jpg)
Redis Top Differentiators
4
Simplicity Extensibility PerformanceNoSQL Benchmark
1
Redis Data Structures
2 3
Redis Modules
Lists
Hashes
Bitmaps
Strings
Bit field
Streams
Hyperloglog
Sorted Sets
Sets
Geospatial Indexes
![Page 5: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/5.jpg)
5
Data Stream Use Cases
![Page 6: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/6.jpg)
6
What is a Data Stream?
![Page 7: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/7.jpg)
Simplest form of a Data Stream
7
ConsumerProducer
![Page 8: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/8.jpg)
But Often You Have M Producers and N Consumers
8
Producer 2
Producer m
Producer 1
Producer 3
Consumer 1
Consumer n
Consumer 2
Consumer 3
![Page 9: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/9.jpg)
It’s a bit more complex then that
9
![Page 10: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/10.jpg)
Data Stream Components
10
Ingest StreamMessage Broker
orStream Processor
Output Stream Analytics
Collect large volume of data arriving in high velocity
Store the data temporarily
Transform incoming data into one or more formats as required by the consumers
Push the data to the consumers
Allow backward looking queries to pull data
Store the data and perform analytical operations such as predictive analytics, ML training, ML classification and categorization, recommendations, pattern matching, etc.
IoT
Activity logs
Messages
![Page 11: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/11.jpg)
11
![Page 12: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/12.jpg)
Redis Supports All the Functions of a Data Stream
12
Ingest StreamMessage Broker
orStream Processor
Output Stream Analytics
Redis
• Streams • Pub/Sub• Lists
• Streams• Pub/Sub• Lists
• Streams• Sorted Sets
+• Spark Connector• ODBC/JDBC Connector
• Sorted Sets• Sets• Geo• Bitfield
+• RedisGraph• RediSearch
![Page 13: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/13.jpg)
13
![Page 14: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/14.jpg)
Understanding Redis Streams
14
![Page 15: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/15.jpg)
Let’s look at the challenges first
15
![Page 16: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/16.jpg)
How to Support a Variety of Consumers
16
Analytics
Data Backup
Consumers
Producer
Messaging Real-time
Real-time or Periodic Lookup
Periodic Read
![Page 17: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/17.jpg)
The Catch Up Problem
17
Image Processor Producer
Consumption Rate: 100/secArrival Rate: 500/secRedis Stream
Fast Slow
Backlog
![Page 18: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/18.jpg)
How to Address The Catch Up Problem?
18
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
Scale Out
New Problem: Mutual Exclusivity
![Page 19: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/19.jpg)
Data Recovery from Consumer Failures
19
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
Scale Out
Yet Another Problem: Failure Scenarios
![Page 20: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/20.jpg)
Connection Failures
20
Producer
Image Processor
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
The solution must be resilient to failures
![Page 21: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/21.jpg)
Here comes Redis Streams
21
![Page 22: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/22.jpg)
What is Redis Streams?
22
Pub/Sub Lists Sorted Sets
It is like Pub/Sub, but with persistence
It is like Lists, but decouples producers and consumers
It is like Sorted Sets, but asynchronous
+• Lifecycle management of streaming data• Built-in support for timeseries data• A rich choice of options to the consumers to read streaming and static data• Super fast lookback queries powered by radix trees
• Automatic eviction of data based on the upper limit
![Page 23: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/23.jpg)
1. It enables asynchronous data exchange between producers and consumers
23
Messaging Producer
Consumer
![Page 24: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/24.jpg)
24
Analytics
Data Backup
Consumers
Producer
Messaging
2. You can consume data in real-time as it arrives or lookup historical data
![Page 25: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/25.jpg)
25
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Image Processor
Redis Stream
3. With consumer groups, you can scale out and avoid backlogs
![Page 26: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/26.jpg)
26
Classifier 1
Classifier 2
Classifier n
Consumer Group
XREADGROUP
XREAD
Consumers
Producer 2
Producer m
Producer 1
Producer 3
XADDXACK
Deep Learning-basedClassification
Analytics
Data Backup
Messaging
4. Simplify data collection, processing and distribution to support complex scenarios
![Page 27: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/27.jpg)
Quick Reference Guide
https://redislabs.com/docs/getting-started-redis-streams/
Redis Streams Commands
27
![Page 28: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/28.jpg)
Commands: XADD, XREAD
28
Asynchronous producer-consumer message transfer
Messaging Producer
ConsumerXADD XREAD
XADD mystream * name Anna XADD mystream * name BertXADD mystream * name Cathy
XREAD COUNT 100 STREAMS mystream 0
XREAD BLOCK 10000 STREAMS mystream $
![Page 29: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/29.jpg)
Commands: XRANGE, XREVRANGE
29
Lookup historical data
Analytics
Data Backup
Consumers
Producer
Messaging
XADD
XREADXRANGEXREVRANGE
XRANGE mystream 1518951123450-0 1518951123460-0 COUNT 10
XRANGE mystream - + COUNT 10
![Page 30: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/30.jpg)
Scaling out using consumer groups
30
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Redis Stream
![Page 31: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/31.jpg)
Scaling out using consumer groups
31
Producer
Image Processor
Arrival Rate: 500/sec
Consumption Rate: 500/sec
Image Processor
Image Processor
Image Processor
Redis Stream
Consumer Group
Consumer 1
Consumer 2
Consumer 3
Consumer n
Unconsumed List
Consumers
XADD XGROUP XREADGROUPXACKXPENDINGXCLAIM
![Page 32: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/32.jpg)
Command: XGROUP CREATE
32
Create a consumer group
Unconsumed List
mygroup
Alice
Bob
edcba
mystream
edcbaApp A
App B
XGROUP CREATE
XGROUP CREATE mystream mygroup $ MKSTREAM
![Page 33: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/33.jpg)
Command: XREADGROUP
33
Read the data
Unconsumed List
mygroup
Alice
Bob
App A
App B
aed
cb
XREADGROUP
XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >
XREADGROUP GROUP mygroup COUNT 2 Bob STREAMS mystream >
![Page 34: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/34.jpg)
Command: XACK
34
Consumers acknowledge that they consumed the data
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
cb
XACK
XACK mystream mygroup 1526569411111-0 1526569411112-0
![Page 35: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/35.jpg)
Command: XREADGROUP
35
Repeat the cycle
Unconsumed List
mygroup
Alice
Bob
App A
App B
a
cb
XREADGROUP
XREADGROUP GROUP mygroup COUNT 2 Alice STREAMS mystream >
![Page 36: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/36.jpg)
How to claim the data from a consumer that failed while processing the data?
36
Unconsumed List
mygroup
Alice
Bob
App A
App Bcb
Unconsumed List
mygroup
Alice
Bob
App A
App B
cb
XCLAIM
![Page 37: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/37.jpg)
Commands: XPENDING, XCLAIM
37
Claim pending data from other consumers
Unconsumed List
mygroup
Alice
Bob
App A
App B
cb
XPENDING mystream mygroup - + 10 Bob
XCLAIM mystream mygroup Alice 0 1526569411113-0 1526569411114-0
![Page 38: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/38.jpg)
Now is the time to
38
![Page 39: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/39.jpg)
Sample Use Case: Social Media Analytics
39
a
b
x
Language classifier
XREADGROUP
Real-time reports
XREAD
Sentiment analyzers Consumer is x times slower than the producer
Location classifier
Influencer classifierRedis Streams
![Page 40: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/40.jpg)
Demo
40
Redis Streams
Twitter Ingest Stream Influencer Classifier
https://github.com/redislabsdemo/IngestRedisStreams
![Page 41: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/41.jpg)
Java Jedis https://github.com/xetorthio/jedis
Lettuce https://lettuce.io/
Python redis-py https://github.com/andymccurdy/redis-py
JavaScript ioredis https://github.com/luin/ioredis
node_redis https://github.com/NodeRedis/node_redis
.NET StackExchange.Redis https://github.com/StackExchange/StackExchange.Redis
Go redigo https://github.com/gomodule/redigo
go-redis https://github.com/go-redis/redis
C Hiredis https://github.com/redis/hiredis
PHP predis https://github.com/nrk/predis
phpredis https://github.com/phpredis/phpredis
Ruby redis-rb https://github.com/redis/redis-rb
Redis Clients that Support Redis Streams
41
![Page 42: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/42.jpg)
Questions
??
?
?
?
?
??
?
??
![Page 43: Intro to Redis Streams · Who We Are 3 Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source](https://reader034.vdocuments.us/reader034/viewer/2022043022/5f3e131650ef3a759c3f6e37/html5/thumbnails/43.jpg)
Thank you!redislabs.com
43