redis and python at pycon_2011
DESCRIPTION
These is the slide deck which I presented in tutorial section at PyCon India in Pune (India) in 2011.TRANSCRIPT
![Page 1: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/1.jpg)
Redis And Python
PyCon India, 2011Sunil Arora
![Page 2: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/2.jpg)
Raising Hands...
How many of you have used Redis before ?
How many of you have got a laptop ?
![Page 3: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/3.jpg)
About Me
I tweet at Sunil Arora / @_sunil_
I work at ShopSocially
I blog at http://sunilarora.org
![Page 4: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/4.jpg)
Today's talk
What is Redis
How it works and what you can do with it
Real life use-casesReal life use-cases
Some hand-on with Redis
![Page 5: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/5.jpg)
Redis - Brief History
Initially written to improve performance of Web Analytics product LLOOGG out of his startup
Salvatore Sanfilippo
@antirez
http://antirez.com
![Page 6: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/6.jpg)
Redis – Brief History
Released in March 2009 (Open Source, BSD licensed)
VMWare hired Salvatore in March, 2010
Then Pieter Noordhuis (key contributor) was hired
![Page 7: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/7.jpg)
What is Redis ?
Its between lot of stuff, so difficult to categorize it precisely
![Page 8: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/8.jpg)
What is Redis ?
I see Redis definitely more as a flexible tool that as a
solution specialized to solve a specific problem: his mixed soul of cache,
store, and messaging server shows this very
well
-Salvatore Sanfilippo
Picture by herzogbrhttp://www.flickr.com/photos/herzogbr/2274372747/sizes/z/in/photostream/
![Page 9: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/9.jpg)
Redis is...
Remote Data Structure Server
![Page 10: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/10.jpg)
Redis is...
Supports rich data types of computer science
- Strings, Lists, Sets, Sorted Sets, Hashes...
Rich sets of primitives (commands) to manipulate these types
Predictive complexity measurements
![Page 11: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/11.jpg)
A few fundamentals
Written in C (no external dependency) Uses memory as main storage Uses disk for persistence Single Threaded Every command performs atomic execution
![Page 12: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/12.jpg)
Performance
Screamingly fast performance ~50K read/write operations per seconds 100K+ read/write ops per second on a regular
EC2 instance
![Page 13: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/13.jpg)
Installation
$ git clone http://github.com/antirez/redis
$ cd redis
$ make
$ ./src/redis-server
..........
..........
$./src/redis-cli
Redis> PING
PONG
![Page 14: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/14.jpg)
Python Libraries
Redis-py - Python client library
Txredisapi - An asynchronous Python client for the Redis database, based on Twisted.
Redisco - ORM for redis along the lines Ohm for Ruby
![Page 15: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/15.jpg)
redis-py
The most popular python client library Andy McCurdy ([email protected]) Github: http://github.com/andymccurdy/redis-py
$easy_install redis OR
$pip install redis Optional
$easy_install hiredis
$pip install hiredis
![Page 16: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/16.jpg)
Lets get started...
$ cd redis
$ ./src/redis-server
.........
>>> from redis import Redis
>>> redis_client = Redis()
>>> redis_client.keys()
>>> help(redis_client)
![Page 17: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/17.jpg)
Redis Keys
Not binary safe. Should not contain space or newline character A few rules about keys:
Too long keys are not a good idea Too short keys is also not a good idea “object-type:id:field” can be a nice idea, i.e.
“user:1001:name”
![Page 18: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/18.jpg)
Operations on Keys
KEYS EXISTS DEL EXPIRE OBJECT PERSIST
RANDOMKEY RENAME TYPE TTL EXPIREAT MOVE
![Page 19: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/19.jpg)
Lets play with keys
>>>redis_client.keys()
>>>redis_client.exists('key')
>>>redis_client.delete('key')
>>>redis_client.type('key')
>>>......
>>>......
![Page 20: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/20.jpg)
Data Structures
Strings Lists Sets Sorted Sets Hashes
![Page 21: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/21.jpg)
Strings
SET GET MSET MGET SETEX GETSET SETNX
INCR INCRBY DECR DECRBY
![Page 22: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/22.jpg)
Strings – with redis client
>>> redis_client.set('key', 'value')
>>> redis_client.get('key')
>>> redis_client.delete('key')
![Page 23: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/23.jpg)
Fetch multiple keys at once
mget/mset redis_client.mset({'key1': 'val1', 'key2': 'val2'}) redis_client.mget('key1', 'key2',......)
![Page 24: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/24.jpg)
Expiration
Set a value with expire
>>>redis_client.setex('key', 'value', 2) #key to expire in 2 secs
>>>redis_client.expire('key', 2)
>>>redis_client.get('key')
>>>None
![Page 25: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/25.jpg)
Expire Semantics
Key with expiry known as volatile keys
Whenever a volatile key is modified, its expiry is reset
- To avoid inconsistency in following cases• Replication
• Append Only Log
![Page 26: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/26.jpg)
Uses
To store transient states in your web application
![Page 27: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/27.jpg)
Uses
Who is online?
![Page 28: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/28.jpg)
Uses
Redis as LRU cache (http://antirez.com/post/redis-as-LRU-cache.html)
![Page 29: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/29.jpg)
Atomic Increments
>>>help(redis_client.incr)
>>>help(redis_client.decr)
>>>
>>> redis_client.incr('counter', 1)
>>>1
>>> redis_client.incr('counter')
>>>2
>>> redis_client.incr('counter')
>>>3
![Page 30: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/30.jpg)
Uses
High Speed counters (views/clicks/votes/likes..)
![Page 31: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/31.jpg)
Uses
API Rate Limiting
![Page 32: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/32.jpg)
Uses
Generating unique IDs
![Page 33: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/33.jpg)
Lists
Ordered list of binarysafe strings Doubly linked list Memory footprint optimized for smaller list O(1) insertion/deletion at both ends
![Page 34: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/34.jpg)
Lists - operations
LPUSH RPUSH LSET LRANGE LPOP BLPOP BRPOP BRPOPLPUSH
LINSERT RPOP RPOPLPUSH LPUSHX RPUSHX
![Page 35: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/35.jpg)
Uses
Web apps are full of lists :)
![Page 36: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/36.jpg)
Uses
Capped List
![Page 37: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/37.jpg)
Uses
Real time message QueueBackground Worker queues (Resque, Celery)
https://github.com/blog/542-introducing-resque
![Page 38: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/38.jpg)
Uses
Social Activity Streams or notifications
![Page 39: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/39.jpg)
Sets
An unordered collection of distinct byte strings
Nothing different from the data type in python
![Page 40: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/40.jpg)
Sets - Operations
SADD SCARD SREM SISMEMBER SMEMBERS SPOP SRANDMEMBER
![Page 41: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/41.jpg)
Operations between Sets
SMOVE SUNION SDIFF SINTER
![Page 42: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/42.jpg)
Sets - Operations
SDIFFSTORE SINTERSTORE SUNIONSTORE
![Page 43: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/43.jpg)
Sets - Uses
Picking random items from a set using SRANDMEMBER
Ex.
– Picking a random article from daily news
– Pick a random Ad for serving
– Pick a random option for A/B
Note: Time complexity is O(1). Compare it with SQL's “order by Rand()”
![Page 44: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/44.jpg)
Sets - Uses
To model Relations in social graph Ex.
>>>redis_client.sadd('friends:john', 'jenny', 'maria')
>>>redis_client.sadd('friends:ben', 'maria', 'kate')
>>>redis_client.sinter('friends:john', 'friends:ben')
![Page 45: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/45.jpg)
Relations (friend/followers)
Common followlist for A and B
>>>sinter('users:A:follows', 'users:B:follows')
Unique to B compared to C
>>>sdiff('users:B:follows', 'users:C:follows') Mutual relationship (friend as well as follower)
>>>sinter('users:B:followers', 'users:B:friends')
Who does not follow me back ?>>>sdiff('users:B:friends', 'users:B:followers')
Who am I not following back?>>>sdiff('users:B:followers', 'user:B:friends')
![Page 46: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/46.jpg)
Which of my friend's are online right now?
http://www.lukemelia.com/blog/archives/2010/01/17/redis-in-practice-whos-online/
SINTER online_people my_friends
![Page 47: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/47.jpg)
Which of my friends are online right now?
>>>RENAME online:fresh online:stale #every minute
>>>SUNIONSTORE online:fresh online:stale
>>>#friend jack connects
>>>SADD online:fresh 'jack'
>>>SUNIONSTORE online online:fresh online:stale
>>>#who is online ?
>>>SINTER online friends
![Page 48: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/48.jpg)
Sorted Sets
Ordered sets on the basis of score
![Page 49: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/49.jpg)
Sorted Sets
ZADD ZCARD ZCOUNT ZINCRBY ZINTERSTORE ZRANGE ZRANGEBYSCORE ZRANK
ZREM ZREMRANGEBYRA
NK ZREMRANGEBYSC
ORE ZREVRANGE ZREVRANGEBYSCO
RE ZSCORE ZUNIONSTORE
![Page 50: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/50.jpg)
Sorted Sets – Use Case
To build index on your dataset
![Page 51: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/51.jpg)
Uses - Realtime Leaderboards
>>>zincrby('leaderboard', 'john', 2)
>>>zincrby('leaderboard', 'jack', 5)
>>>zincrby('leaderboard', 'kate', 1)
>>>zincrby('leaderboard', 'kate', 10)
....
>>>zrange('leaderboard', 0, -1, withscores = True)
![Page 52: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/52.jpg)
Uses - Realtime Leaderboards
Most download resources
Most popular articles on the website
Weekly popular list
Most downloaded stuff between date1 and date2
.....
.....
![Page 53: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/53.jpg)
AutoComplete with Sored Sets
http://antirez.com/post/autocomplete-with-redis.html
![Page 54: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/54.jpg)
Hashes
Equivalent to Python dictionary or Ruby hash or Java hashmap
Operations– hmset users:1 {'username': 'jim', 'score': 23}
– hgetall users:1
– Hincrby Useful for storing structured data
– hmset user:1:preferences {'flash_shown': 'yes', 'bgcolor': '#fff'}
– Expire user:1:preferences <duration>
– Hmgetall user:1:preferences #returns entire dict
![Page 55: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/55.jpg)
Redis Transactions
Using Multi/Watch
>>>p = redis_client.pipeline()
>>>p.lpush('a', 1)
>>>p.ltrim('a', 0, 100)
>>>p.execute Limitation
– Since commands are queued, Can't read values
– No conditional execution
– If not high write contention scenario, WATCH can be used
Use Redis Scripting to write your own primitive
![Page 56: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/56.jpg)
Designing with Redis
Data layout design on basis of Query
No Query Optimizer
Manually build indexes
![Page 57: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/57.jpg)
Durability
Snapshotting mode
– Binary dump every x secs or y ops Append Only File (AOF)
– Every command is written to a file
– On restart/crash, commands replayed
– Fsync on every new command
– Fsync every second
– OS decide to Replication
![Page 58: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/58.jpg)
Publish/Subscribe
A simple and efficient implementation of publish/subscribe messaging paradigm
Client can subscribe/psubscribe to receive messages on channels (key patterns)
![Page 59: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/59.jpg)
Publish/Subscribe
PSUBSCRIBE PUBLISH PUNSUBSCRIBE SUBSCRIBE UNSUBSCRIBE
![Page 60: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/60.jpg)
Uses
Many to Many message passing
![Page 61: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/61.jpg)
Uses
Web Chat
![Page 62: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/62.jpg)
References
Redis Website (http://redis.io) SimonWillison Tutorial (
http://simonwillison.net/static/2010/redis-tutorial/) Redis from ground up (
http://blog.mjrusso.com/2010/10/17/redis-from-the-ground-up.html#heading_toc_j_19)
Redis under the hood (http://pauladamsmith.com/articles/redis-under-the-hood.html)
Tumbler and Redis (http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-notifications)
![Page 63: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/63.jpg)
References
Redis at disqus (http://bretthoerner.com/2011/2/21/redis-at-disqus/)
Redis at Craiglist http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigslist/
Redis at Bump
http://devblog.bu.mp/how-we-use-redis-at-bump
Redis: AK 47 of postrelational database
http://www.slideshare.net/karmi/redis-the-ak47-of-postrelational-databases
![Page 64: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/64.jpg)
Questions
![Page 65: Redis And python at pycon_2011](https://reader035.vdocuments.us/reader035/viewer/2022062219/554a25d1b4c90542548b4a21/html5/thumbnails/65.jpg)
Thanks
Contact details
Twitter: @_sunil_Email: arora.sunil AT gmail.com