por que deberias haberle pedido redis a los reyes magos

javier ramirez@supercoco9

Por qu deber as haberle é ípedido Redis a los Reyes Magos

In the beginner's mind there are many possibilities, in the expert's mind there are few.

Shunryu Suzuki

javier ramirez @supercoco9 https://teowaki.com madrid rb 2013

https://teowaki.com/

myth: the bottleneck


Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (with pipelining)

$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q

SET: 552,028 requests per secondGET: 707,463 requests per secondLPUSH: 767,459 requests per secondLPOP: 770,119 requests per second

Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (without pipelining)$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q

SET: 122,556 requests per secondGET: 123,601 requests per secondLPUSH: 136,752 requests per secondLPOP: 132,424 requests per second



myth: Redis is just like memcached



myth: Redis is for queues



open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.

http://redis.io

started in 2009 by Salvatore Sanfilippo @antirez

103 contributors at https://github.com/antirez/redis



hacker meets thinker*

*@janogonzalez and @advi inspired slide



The Redis Manifesto1.A DSL for Abstract Data Types

2.Memory storage is #1

3.Fundamental data structures for a fundamental API

4.Two levels of API

5.Code is like a poem; it's not just something we write to reach some practical result

6.We're against complexity

7.We optimize for joyjavier ramirez @supercoco9 https://teowaki.com madrid rb 2013


connecting to redis


basics: setting and getting


basics: lists


basics: hashes


basics: sets

cool example about sunionstore

http://robots.thoughtbot.com/post/46335890055/redis-set-intersection-using-sets-to-filter-data


Redis as a PUBSUB system


http://robots.thoughtbot.com/post/46335890055/redis-set-intersection-using-sets-to-filter-data

Redis keeps everything in memory all the time


Persistance: RDB Compact binary formatSaves snapshots every few minutesGood for backups and synchronizing

If Redis crashes, a few minutes worth of data will be lost



Durability: AOF

Log text formatConfigurable durabilityLarge file, can slow down startup

If Redis crashes, typically one second of data could be lost



replication & scaling out

one master, several read-only slaves

sharding

Twemproxy & redis clusterjavier ramirez @supercoco9 https://teowaki.com madrid rb 2013


atomicity

single threaded, so no concurrency problems

transactions and lua scripts to run multiple operations atomically



Scripting with luaYou can use Lua for scripting Redis when you need to atomically execute a sequence of commands in which the output of a command is used as input for another

It reduces the need to use complex lock mechanisms and simplifies dependencies between clients

You can even extend the functionality of Redis by using Lua scripts



what's being used for


twitter

Every time line (800 tweets per user) is on redis

5000 writes per second avg300K reads per second


twitter

user info from gizmoduck(memcached)

user id tweet id metadata

write API (from browser or client app)

rpushx to Redis

tweet info from tweetypie (memcached + mysql)

your twitter timeline


fanout (flockDB)one per follower

pinterest object graph

from mysql+memcached to redis

30% save on IOps


per userRedis SortedSet, with timestamp as the score, is used to store the users followed explicitly. Redis SortedSet, with timestamp as the score, is used to store the users followed implicitlyRedis SortedSet, with timestamp as the score, is used to store the user’s explicit followersRedis SortedSet, with timestamp as the score, is used to store the user’s implicit followersRedis Set is used to store boards followed explicitlyRedis Set is used to store boards unfollowed explicitly

per boardRedis Hash is used to store a board’s explicit followersRedis Set is used to store a board’s explicit unfollowers


viacomObject dependency graph. Cache on steroids

Redis as a queue for background jobs

Activity tracking and view counts buffering before saving to mysql

Lua script working on a slave to recalculate ranking and popularity of contents, then send the data to master. The new process takes 1/60th less than the old one in mysql


openredis

redis as a service on AWS

serving over 1000 customers with a single machine


when things go wrong


the instagram case

moving from redis to cassandra: 75% savings on servers

lesson learnt: know when redis is not appropriate


the twilio case

credit card hell

lesson learnt: know what you are doing. Don't change config on the fly


how teowaki is using redis


Abusing sidekiq/resque

keep track of every activity in the system, even if you don't need them all right now:- every page view- every API request- every time a record is created/updated/deleted

benefits:- highly decoupled system- easier to divide into services- you can add behaviour without changing your app


intermediate cache* As a very fast lightweight storage for analytics data before sending them to our google bigquery based solution

* As a cache for AR attributes frequently “plucked” or includedand plucked (names, nicknames, guids, delegated or included model names...)


Some of our uses of LuaExpiring attributes inside a Redis hash

Inserting notifications into a list only if there are notpending notifications from the same user for the same scope

Paginating a list by an attribute

Manipulating JSON directly at the Redis layer


counters

Atomic counters can be safely invoked concurrently from anywhere, so you can implement “like” features, global sequences or usage monitoring systems in highly concurrent applications for free.

You can share your counters with any other internal application and still be sure they won't collide.


Temporary dataRedis allows us to self expire keys after a time has passed. You can use this mechanism for a simple cache

If you operate on a key with expiration time, you can change its value and still keep the expiration going. By combining a decrementing counter with an expiration time, implementing usage quotas is trivial

Also, you can inspect which keys you have in your server efficiently using SCAN commands


bloom filtersbloom filter: space-efficient probabilistic data structure that is used to test whether an element is a member of a set. False positive matches are possible, but false negatives are not.

Redis bit operations make easy to implement bloom filters

We are using bloom filters for checking uniqueness of user names and reserved words without going to postgresql


nginx + lua + redisapache + mruby + redis

Multiple levels of cache by using Redis on the webserver/middleware layer

http://wiki.nginx.org/HttpRedishttps://github.com/jodosha/redis-store


summarizing* Redis is more powerful than it seems

* Very fast, easy to use, simple, good documentation

* In-memory data structures, distributed, shared and persisted

* Good as data store, intermediate data store, cache or queue

* Lots of use cases, both in huge and smaller systems

You should probably use it a lot more


Find related links at

https://teowaki.com/teams/javier-community/link-categories/redis

Gracias!¡

Javier Ramírez@supercoco9

madrid rb 2013

por que deberias haberle pedido redis a los reyes magos

Technology

commadrid rb

second javier ramirez

lostjavier ramirez

queuesjavier ramirez

forjavier ramirez

bottleneckjavier ramirez

gettingjavier ramirez

listsjavier ramirez