REDIS – Advanced Key Value Store

Rajan Bhatt

What REDIS is ?

• REDIS is an open source, BSD licensed, advanced key-value store. It is often referred to as data structure server since keys can contain String, Hashes, List, Sets and Sorted sets..

• REDIS stands for REmote DIctionary Server

REDIS - Features

• Simplicity• Speed• Low Footprint• Versatility • Predictability• Reliability

How to Install and Start REDIS server, REDIS client and Bindings

• Go to http://redis.io/download and download• Start Redis server ( TCP server, Single

Threaded, Listens on default port 6379 ).• Command – redis-server <path to conf. file )• Bundled REDIS client – redis-cli • Client bindings available for all popular

languages, Java, Python, Ruby, Scala, Ruby, Eralang, Go ..( I am running out of space )

Data Structure

• Strings• List• Sets• Sorted Set• Hash• And Publish/Subscribe Design Pattern

Strings

• Set Key “Value” ( 4 billion Keys ) • Get Key returns “Value”• Del Key• Fetch Multiple Keys at once … MGET• Set EXPIRATION key, e.g. Expire Key 5 • Command TTL returns remaining time before

key expires

Use Case for Expire, TTL

• Cache – LRU http://antirez.com/post/redis-as-LRU-cache.html

• Storage for Session ( HTTP or others ..)

Atomic Counter

• GET Key Nil • INCR Key1• INCR Key 2• Get Key 2

Usage of INCR

• Great Use case Global Counters ( Download, Hits, Votes ..)

• INCR downloads:total • INCR downloads:total:today • INCR downloads:total:2011-05-10 • INCR downloads:/downloads/file1.mpg:total • INCR downloads:/downloads/file1.mpg:today • INCR downloads:/downloads/file1.mpg:2011-05-

10

Counters

# Total downloads for server, all time • GET downloads:total # Total downloads for server, today • GET downloads:total:today # Total downloads for file • GET downloads:/downloads/file1.mpg:total # Total downloads for file today • GET downloads:/downloads/file1.mpg:today

How do you reset daily counter ?

# Expire at 2011-05-10 23:59:59

• EXPIREAT downloads:total:today 1305064799

One more Use Case – API rate Limiting

• $ curl http://api.example.com/list.json

• INCR api:<TOKEN>:hits1

• Pseudo-Code if INCR('api:abc123:hits') > LIMIT return 420 Enhance Your Calm end

One more Use case Generating unique IDs

• INCR global:users_ids1• SET users:1:username "john" • INCR global:users_ids2• SET users:2:username "mary”

List• LPUSH key 1 1• LPUSH Key 2 2• LPUSH Key 3 3 • RPOP Key What should I get ?• LPOP Key What should I get ?• LLEN Key ( Length if List )• LRANGE key 0 -1 • LTRIM 0 1

Usage of List

• Indexes ( List of comments ..)LPUSH article:comments <ID>Timelines (of all sorts: messages, logs, ...)

LPUSH user:<ID>:inbox "message from Alice" LPUSH user:<ID>:inbox "message from Bob"

# Limit the messages to 100LTRIM user:<ID>:inbox 0 99# Get last 10 messagesLRANGE user:<ID>:inbox 0 9# Get next 10 messagesLRANGE user:<ID>:inbox 10 19

List is “Queue”

Publisher • RPUSH queue “task-1”• RPUSH queue “task-2”

Worker ( blocks and waits for tasks, 0 means waits for ever )

• BLPOP queue 0

Basic Set

• SADD key 1 1 • SADD key 2 2• SADD key 3 3• SMEMBERS key “3”, “2”, “1”• SISMEMBER key 1 “1”• SISMEMBER key 5 “0”• SREM key 3 1

Set Operations

• SADD A 1 • SADD A 2 • SMEMBERS A “1”, “2”• SADD B 1• SADD B 3• SMEMBERS “1”, “3”

Set Continues..

• SUNION A B “1”, “2”, “3”• SINTER A B “1”• SDIFF A B “2”

Set Usages - Relationship

• SADD users:A:follows B • SADD users:B:follows C • SADD users:B:follows D • SADD users:C:follows A • SADD users:C:follows D

Set Relationship

• Joint network of A & B SUNION users:A:follows users:B:follows “C”,”D”,”B”• Common for A & B SINTER users:B:follows users:C:follows []• Unique to B compared to C “C”

Set – Friends & Followers# Whom “swid1” follows• SADD swid1:follows A• SADD swid1:follows B # Who are friends of “swid1” • SADD swid1:friend C• SADD swid1:friend B# Whom swid1 follows and he is friend• SINTER swid1:follows swid1:friend “B”# Whom swid1 follows and he is not friend• SDIFF swid1:follows swid1:friend “A”# Whom Swid1 is friend with but does not follow• SDIFF swid1:friend swid1:follows “C”

Sorted Set

• ZADD key 100 A• ZADD key 10 C• ZADD key 80 B• ZRANGE key 0 -1 “C”, “B”, “A”• ZREVANGE key 0 -1 “A”, “B”, “C”• ZINCRBY key 10 C “20”

Sorted Set - Continues

• ZREVRANGE key 0 -1 WITHSCORES 1) "A" 2) "100" 3) "B" 4) "80" 5) "C" 6) "20" • ZREVRANGEBYSCORE key 100 50 • 1) "A" • 2) "B"

Leaderboards

# User A got 10 pointsZINCRBY scores 10 A# User B got 15 points ZINCRBY scores 15 B# User A got another 10 points ZINCRBY scores 10 A # Display scores ZREVRANGE scores 0 -1 WITHSCORES 1) "A" 2) "20" 3) "B" 4) "15"

Hashes

• Redis Hashes are maps string filed and string values.

• This data type is perfect to represent an object.

• A hash with few fields ( 100s ) is stored in a way that takes very little space so you can store millions of object is small REDIS instances.

REDIS Hash commands

• HMSET user:1001 username john cid 1234 dept finance eaccount 5000

• HMGET user:1001 username cid ( Multi-field GETs )

• HINCRBY user:1001 eaccount 5 ( Incrementing specific field )

• HGET user:1001 eaccount ( Get specific field )• HSETNX user:1001 username

REDIS - Persistence

• All datasets are stored in Memory like Memcached, extremely fast read/write operations.

• Datasets can be saved to Disk, Two ways– RDB – AOF ( Append only File )

RDB snapshots and AOF logs

• Persistence REDIS is a matter of configuration, balancing the trade-off between performance, disk I/O and data durability.– RDB is a very compact single-file point-in-time

representation of REDIS dataset.– AOF is a simple text log of write operations.

REDIS HA

• REDIS support Master/Slave configuration for HA• Master can have multiple Slaves and Slaves can also be

connected other slaves in Graph like structure.• REDIS replication is non-blocking on Master and Slave side.• Replication is used both for scalability or data redundancy.• To configure Slave is very simple just needs to add following to

slave configuration file, Slaveof <ip-addr-of-server> 6379 • By default slaves are read only.• What if Master fails ? How do client behave ? What are

options ?

Redis Scalability

• Client Side Sharding – Client shards key based on number of Redis servers. This is static partitioning. Each Server is configured with Master/Slave for redundancy.

• Server Side Sharding – twemproxy (https://github.com/twitter/twemproxy) by Tweeter, Consistent Hashing implementation. Another resource http://antirez.com/news/44.

• Redis sever Pre-Sharding strategy - e.g Pinterest • Great resource to start - http://redis.io/topics/partitioning.• Many options.. Roll your Own which meets needs..

How to Monitor REDIS ?

• Local Monitoring : Monitd to monitor REDIS instance locally. Restart if process fails. AS team has prototype for this.

• Remote Monitoring and Integration with Zenoss - https://community.zenoss.org/docs/DOC-5333.

• Accessing REDIS metrics – Start redis-cli – info command with following options, server, clients, memory, persistence, stats, replication, cpu, commandstats, cluster,keyspace

• Many commercial SAAS vendors, DATADOG, LIBRATO, more analysis and statistical Metrics collection.

REDIS as a Service

• Redislabs - http://redislabs.com/• Amazon Elastic Cache service -

http://aws.amazon.com/about-aws/whats-new/2013/09/04/amazon-elasticache-for-redis/

How to Measure REDIS performance ?

• Redis-benchmark is a tool which comes bundled with Redis installation.

• Check out at http://redis.io/topics/benchmarks.

• This tool simulates N client sending M queries.• Let us run.. See what happens.

Other feature of REDIS

• Transactions – MULTI,EXEC,DISCARD AND WATCH are the transactions in Redis.– Redis transactions are atomic.– Redis transactions are started with MULTI command and then all

commands are executed once EXEC is executed.

• LUA scripting – Lua scripting is used to implement custom command and/or

extension. This is very similar to stored procedure in RDBMS.– Redis Lua interpreter loads seven libraries:

base,string,table,math,debug,cjson and cmsgpack.

Summary

• Redis offers excellent caching and persistence options along with various data structure for applications.

• This is true Lego building block for a service developer. Use case is only limited by imagination.

• At least start with, replacing Memcached.• Excellent Live Documentation..check it out

http://redis.io

Q&A