redis si resque
DESCRIPTION
Redis and Resque presentation at Cluj.rb File with Demo script: https://gist.github.com/florin555/5012229TRANSCRIPT
![Page 1: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/1.jpg)
Redis si Resquepentru applicatii Rails concurente
Florin OlteanCluj.rb
2013-02-21
![Page 2: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/2.jpg)
Despre mine
• Lucrez la ZenCash
• sincronizare cu servicii externe
• monitorizarea aplicatiei
![Page 3: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/3.jpg)
Cuprins
• Redis
• Resque
• Cum folosim Redis si Resque in ZenCash
![Page 4: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/4.jpg)
![Page 5: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/5.jpg)
Redis
• key-value data store
• networked
• in-memory
• optional durability
• open-source (BSD license)
![Page 6: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/6.jpg)
Calitati
• usor de folosit
• documentatie simpla si usor de inteles
• rapid (fiind in memorie)
![Page 7: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/7.jpg)
Cum se foloseste
• redis-cli
• din Ruby:
• Redis gem
![Page 8: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/8.jpg)
Tipuri de date• “Data structures server”
• Tipul de baza: String
• Structuri de date:
• Lists
• Hashes
• Sets
• Sorted sets
![Page 9: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/9.jpg)
• comenzile sunt atomice
• ex:
• incr, decr
• lpush, lpop
• setnx
![Page 10: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/10.jpg)
DEMO
![Page 11: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/11.jpg)
Expire
• putem sa setam un “expire” pe orice cheie
• e important ca acea cheie sa existe
![Page 12: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/12.jpg)
DEMO
![Page 13: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/13.jpg)
Publish-Subscribe
• subscribe la mai multe cozi
• publish => primesc toti cei subscribed
![Page 14: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/14.jpg)
DEMO
![Page 15: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/15.jpg)
Tranzactii
• toate instructiunile sunt executate in ordine
• nu se ruleaza instructiuni primite de la alti clienti
• atomicitate: totul sau nimic
![Page 16: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/16.jpg)
DEMO
![Page 17: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/17.jpg)
Namespaces
• Gem redis-namespace
![Page 18: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/18.jpg)
DEMO
![Page 19: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/19.jpg)
Pipelining
• la comunicare prin TCP, round-trip time mare comparat cu durata unei instructiuni
• ~1ms prin loopback
• ~100 ms prin Internet
• Solutie: sa folosim pipelining
• Pentru si mai multa viteza: comanda eval pentru a rula Lua scripts (doar din v2.6)
![Page 20: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/20.jpg)
DEMO
![Page 21: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/21.jpg)
Persistenta
• Fiind in memorie, toate datele se pierd daca
• moare procesul
• pana de curent (sau crapa sistemul de operare)
![Page 22: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/22.jpg)
Persistenta
• Solutii:
• RDB point-in-time snapshots
• AOF (Append Only File)
![Page 23: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/23.jpg)
Persistenta
• RDB point-in-time snapshots
• util pentru backup
• fork => fisier scris de procesul fiu
• copy-on-write
• poate bloca procesul pana la 1 secunda
• cele mai recente date se pot pierde
![Page 24: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/24.jpg)
Persistenta• AOF (Append Only File)
• foloseste write(2) si fsync(2)
• fsync: no / every second / every query
• fisier mai mare decat RDB
• scade putin performanta
• rescris automat cand creste prea mult
• tot folosind fork
![Page 25: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/25.jpg)
• Ar trebui folosite amandoua metodele pentru persistenta
• RDB mai bun decat AOF pentru disaster recovery pentru ca este mai compact
![Page 26: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/26.jpg)
Resque
![Page 27: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/27.jpg)
Resque
• cozi de lucru
• inspirat de DelayedJob
• foloseste Redis
![Page 28: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/28.jpg)
Structura Resque
• librarie Ruby
• creare, interogare si procesare de joburi
• Rake task pentru pornirea de workeri
• aplicatie Sinatra pentru monitorizare
• cozi, joburi, workeri, errori
![Page 29: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/29.jpg)
Workeri
• pot rula pe masini diferite
• nu au probleme de memory bloat / "leaks"
• arhitectura parent - child (fork)
![Page 30: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/30.jpg)
Cozi
• sunt persistente
• complexitate O(1)
• push si pop atomic
• pot fi inspectate
• stocheaza joburile ca si packete JSON
![Page 31: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/31.jpg)
De tinut cont
• sa avem grija la ce trimitem ca parametri
• id in loc de obiect intreg
• avantaj: obiect up-to-date
• dezavantaj: poate inca nu se vad schimbarile
• schimbare symbol in string (JSON)
• [:a, :b] se transforma in ["a", "b"]
![Page 32: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/32.jpg)
Interfata web
• informatii despre workeri
• cozile folosite
• continutul cozilor
• statistici (ex. numar de job-uri procesate)
• urmarirea erorilor
![Page 33: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/33.jpg)
Resque
![Page 34: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/34.jpg)
Redis si Resquein ZenCashin ZenCash
![Page 35: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/35.jpg)
ZenCash
• Te ajuta sa fii platit mai repede
• Integrare cu 8 aplicatii de facturare
• import: Invoices, Payments, Customers
• Sincronizare:
• o data pe ora
• on-demand
![Page 36: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/36.jpg)
• In ZenCash folosim Redis pentru
• cozi de lucru (Resque)
• stocare temporara
• sincronizare procese
![Page 37: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/37.jpg)
• comunicare cu third-party API
• trimitere de email
• taxare clienti
• joburi pentru
• monitorizare aplicatie
• generare statistici
• lansare sincronizare periodic
• procesare rezultate sincronizare
Resque
![Page 38: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/38.jpg)
Stocare temporara
![Page 39: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/39.jpg)
Sync Status
• Sincronizarea unei facturi
• Invoice
• Customer
• parsare adresa (daca e nevoie)
![Page 40: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/40.jpg)
DB-less SyncApp
• pentru un sync pot fi necesare mai multe request-uri (ex. paginare)
• se faceau serializari / deserializari in plus
• am renuntat la MySQL
![Page 41: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/41.jpg)
Temporary file store
• scenariu:
• un proces face download de PDF
• alt proces face upload pe S3
• procesele trebuie sa fie pe aceeasi masina ca sa poata accesa fisierul de pe disc
• solutie: stocarea fisierului in Redis
![Page 42: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/42.jpg)
Monitorizare aplicatie
• Salvarea periodica a unor parametri
• Bounded Queue:
• coada cu dimensiune limitata implementata folosind Redis
![Page 43: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/43.jpg)
Sincronizare procese
• Redis Lock
• poate fi folosit pentru a ne asigura ca un singur proces executa o bucata de cod pentru o anumita resursa
• ex: procesare rezultate sincronizare
![Page 44: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/44.jpg)
• Exemplu folosire RedisLock
![Page 45: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/45.jpg)
Rate limitation
• Unele servicii au API usage limits
• Tinem o evidenta a requesturilor facute pentru un anumit user in ultimul minut
![Page 48: Redis si Resque](https://reader034.vdocuments.us/reader034/viewer/2022052602/559ab0691a28ab05698b4788/html5/thumbnails/48.jpg)
?