redis: persistence power

Redis: Persistence PowerNick Quaranto / @qrush / [email protected]

Tuesday, August 10, 2010

What is Redis?

“advanced key-value store”

REmote DIctionary Server

data structures server


YOUR APP


YOUR APP

REDIS


the basics

persist data as you think of it

in memory, sync to disk in background

ridiculously fast

master-slave replication

keys = strings, value = data structures


http://try.redis-db.com


http://try-redis.com

http://try-redis.com

use it: redis-cli

% ./redis-cli SET user:1:name qrushOK

% ./redis-cli GET user:1:name "qrush"


use it: redis-rb% gem install redis% irb -rubygems -rredis

>> $redis = Redis.new=> #<Redis client v2.0.3 connected...

>> $redis.set "user:1:name", "qrush"=> "OK"

>> $redis.get "user:1:name"=> "qrush"


FEATURE SWITCHESSTRINGS

based onhttp://github.com/blog/677http://github.com/bvandenbos/redis_feature_control


http://github.com/blog/677


http://github.com/blog/677-how-we-deploy-new-features

http://github.com/blog/677-how-we-deploy-new-features

Rediswitch.features << :super_secretRediswitch.features << :payment_gatewayRediswitch.features << :twitter

if Rediswitch.enabled?(:twitter) # post to twitterelse # failwhale ahoy!end


begin # take some moneyrescue PaymentGateway::TotallyDown => ohno Rediswitch.disable(:payment_gateway) # notify the troopsend


class Rediswitch def self.enabled?(feature) $redis.exists(feature) end

def self.enable(feature) $redis.incr(feature) end def self.disable(feature) $redis.del(feature) endend


feature switch lessons

the real win: no-deploy configuration

fast enough to be transparent

next step: separate users into buckets with sets

http://github.com/jamesgolick/rollout


RATE LIMITERSTRINGS

soon to be in place athttp://hoptoadapp.com


http://hoptoadapp.com


class Choker def restrict? track count_for > 60 endend


class Choker def count_for $memcache.get(key, true).to_i endend


class Choker def track if !$memcache.get(key, true) $memcache.add(key, "0", 1.minute.from_now, true) end $memcache.incr(key) endend


class Choker def track if !$redis.exists(key) $redis.setex(key, 60, 0) end $redis.incr(key) endend


class Choker def count_for $redis.get(key).to_i endend


rate limiter lessons

expire semantics are changing in redis 2.2

benchmark the crap out of it

could use a sorted set instead of strings


API USAGE LOGGINGSTRINGSSORTED SETS

based offhttp://www.production-hacks.com/2010/07/10/redis-api-access-logger/


http://www.production-hacks.com/2010/07/10/redis-api-access-logger/

http://www.production-hacks.com/2010/07/10/redis-api-access-logger/

# one way to do itclass ActionHit < ActiveRecord::Base # t.string :controller # t.string :action # t.integer :counterend

class UserHit < ActiveRecord::Base # t.string :controller_action # t.integer :user_id # t.integer :counterend


# for all controllers{ "statuses#update" => 1410, "users#create" => 931, "home#index" => 2936}

# users hitting an action{ "101" => 42, "102" => 13, "103" => 34}


class StatusesController < ApplicationController def update $redis.incr "statuses#update" $redis.incr "statuses#update:#{user.id}" endend


class StatusesController < ApplicationController def update key = "statuses#update" $redis.zincrby "actions", 1, key $redis.zincrby "users:#{key}", 1, user.id endend


# hits for a specific user>> $redis.zscore "users:statuses#update", 1001=> 42 # list all the controller actions, sorted>> $redis.zrevrange "actions", 0, -1, :with_scores => true

=> ["home#index", "2936", "statuses#update", "1410", "users#create", "931"]


api usage logging lessons

sorted set = high score list

bad at historical usage, trends

good for a simple heartbeat or pulse


JOB QUEUELISTS

based onhttp://github.com/defunkt/resque




class Staple @queue = :default

def self.perform(post_id, tempfile) # complex image resizing, cropping endend


class Post < ActiveRecord::Base after_save :process_with_stapler

def process_with_stapler Resque.enqueue(Staple, self.id, @tempfile) endend


module Resque extend self

def push(queue, item) redis.rpush "q:#{queue}", encode(item) end

def pop(queue) decode redis.lpop("q:#{queue}") endend


class Resque::Worker def work loop do if job = Resque.pop(queue) job.perform else sleep 5 end end endend


module Resque extend self def bpop(queue) decode redis.blpop("q:#{queue}") endend


class Resque::Worker def work loop do job = Resque.bpop(queue) job.perform end endend


job queue lessons

guaranteed atomic actions, no row locking

blocking commands simplify daemons

many more queue commands in redis itself!


GLOBAL ERRORSSETSMULTI/EXEC

a new feature athttp://hoptoadapp.com




# text :globals, :default => '', :null => false

class Project < ActiveRecord::Base def has_global?(name) @globals ||= globals.gsub(/,/,' ').split @globals.include?(name) endend


# MORE TABLES!!!!

class Global < ActiveRecord::Base belongs_to :projectend

class Project < ActiveRecord::Base has_many :globalsend


# Project#global_errors

["MySQL::Error", "MemCache::Error", "Net::HTTPFatalError"]


class Project < ActiveRecord::Base def global_key "project-#{id}-globals" end def has_global?(name) $redis.sismember(global_key, name) end end


class Project < ActiveRecord::Base after_save :save_globals

def save_globals $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end endend


[Mysql::Error, MemCache::Error, Net::HTTPFatalError]

SISMEMBER SISMEMBER


[]

DEL


[Mysql::Error]

DEL

SADD


[Mysql::Error, OpenURI::HTTPError]

DEL

SADD

SADD



DEL

SADD

SADD

SISMEMBER

[]



MULTI

EXEC

DEL

SADD

SADDSISMEMBER


class Project < ActiveRecord::Base after_save :save_globals

def save_globals $redis.multi do $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end endend


global error lessons

avoid joins for simple data

consider race conditions

use append-only file (AOF)


MULTIPLAYER NOTEPADPUB/SUB

based onhttp://github.com/laktek/realie


http://github.com/laktek/realie

http://github.com/laktek/realie

# usage: ruby pub.rb room username

data = {"user" => ARGV[1]}loop do msg = STDIN.gets $redis.publish ARGV[0], data.merge('msg' => msg.strip).to_jsonend


# sub.rb

$redis = Redis.new(:timeout => 0)$redis.subscribe('rubyonrails', 'rubymidwest') do |on| on.message do |room, msg| data = JSON.parse(msg) puts "##{room} - [#{data['user']}]: #{data['msg']}" endend


% ruby pub.rb rubymidwest qrushi give up, i hate markdown

% ruby sub.rb#rubymidwest - [qrush]: i give up, i hate markdown#rubyonrails - [railsn00b]: undefined method posts_path? wtf?#rubymidwest - [turbage]: seriously.


multiplayer notepad lessons

combine with other data structures

can subscribe to channels via patterns

concurrency in ruby is hard

use eventmachine! (or node.js)


more to learn

know your data! (via @antirez)

command reference on the wiki

active IRC, mailing list


AKASENTAI.comredis in the cloud


Thanks!http://redis.io @qrushhttp://rediscookbook.com http://scr.bi/redispower


http://redis.io

http://redis.io

http://rediscookbook.com

http://rediscookbook.com

http://scr.bi/redispower

http://scr.bi/redispower

BONUS ROUND!I prepared way too many examples. Jackpot!


URL SHORTENERSTRINGS

based onhttp://github.com/mattmatt/relink


http://github.com/mattmatt/relink/blob/master/relink.rb

http://github.com/mattmatt/relink/blob/master/relink.rb

require 'sinatra'require 'redis_url'

post '/' do RedisUrl.new(params[:url]).saveend


class RedisUrl attr_accessor :url, :id def initialize(url) @url = url @id = seed # unique string algorithm end

def save $redis.set("relink.url|#{@id}", @url) $redis.set("relink.url.rev|#{@url}", @id) endend


get %r{/(.+)} do |url| u = RedisUrl.find(url) if u u.clicked redirect u.url else status 404 endend


class RedisUrl def self.find(id) u = $redis.get("relink.url|#{id}") if u redis_url = RedisUrl.new(u) redis_url.id = id redis_url end end def clicked $redis.incr("relink.url.clicks|#{@id}") endend


url shortener lessons

common pattern: namespacing

incr/decr assumes value is an integer

wrap behavior into ActiveRecord-like objects

next step: store URLs in a list


LIVEDEBUGGINGLISTS

based onhttp://github.com/quirkey/redisk


http://github.com/quirkey/redisk

http://github.com/quirkey/redisk

def after_save begin # make request to external service rescue Exception => ex logger.error "this shouldn't ever happen!" logger.error ex logger.error ex.backtrace endend


# config/initializers/logger.rb

require 'redisk'path = "#{Rails.env}.log"config.logger = Redisk::Logger.new(path)


class Redisk::IO def write(string) redis.rpush "#{name}:_list", string end def self.readlines(name) redis.lrange("#{name}:_list", 0, -1) endend


live debugging lessons

enables real-time data about your system

dump serialized/marshalled data fast

run the redis instance on a different box

dive deeper: hummingbird


COUNTING DOWNLOADSSTRINGSSORTED SETSHASHES

based onhttp://github.com/rubygems/gemcutter


http://github.com/rubygems/gemcutter

http://github.com/rubygems/gemcutter

# bad idea, dude

class Download < ActiveRecord::Base belongs_to :rubygemend

class Rubygem < ActiveRecord::Base has_many :downloadsend


class Download def self.incr(rubygem) $redis.incr("all") $redis.incr(rubygem) $redis.zincrby("today", 1, rubygem) endend


class Download def self.rollover(version) $redis.rename "today", "yesterday"

dls = Hash[*$redis.zrange("yesterday", 0, -1, :with_scores => true)] dls.each do |key, score| $redis.hincrby key, Date.today, score Rubygem.find_by_name(key).increment!(:downloads, score) end endend


get "/api/v1/downloads/rails.json" do $redis.hgetall("rails").to_jsonend

# returns...{ "2010-07-09" => 1908, "2010-07-10" => 1032, "2010-07-11" => 1091,}


counting downloads lessons

hybrid approach does work!

redis is really not for search

test your migration away from SQL


redis: persistence power

Documents