a 8tracks ama o redis
DESCRIPTION
Na 8tracks a gente acredita na ferramenta certa pra resolver os problemas que a gente encontra pela frente! Passamos por vários desafios, mas encontramos uma ferramenta excelente pra nos ajudar a servir mais de 100 mil request por minuto e tornar nossos problemas de persistência bem mais simples. Redis salvou a nossa pele mais de uma vez!TRANSCRIPT
![Page 1: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/1.jpg)
redis love at 8tracks
@nettofarah
![Page 3: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/3.jpg)
8tracks
![Page 4: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/4.jpg)
8tracks
![Page 5: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/5.jpg)
![Page 6: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/6.jpg)
![Page 7: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/7.jpg)
stats
• users: 13.3mi!
• public playlists: 1.2mi (4,2 mi total)!
• uploaded tracks: 31mi
![Page 8: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/8.jpg)
not a sysadmin talk
![Page 9: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/9.jpg)
getting sudo
http://devopsreactions.tumblr.com/post/83704083404/getting-sudo
![Page 10: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/10.jpg)
a melhor ferramenta para o problema
http://peopletakingpictureswithipads.tumblr.com/
![Page 11: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/11.jpg)
melhor ferramenta
http://peopletakingpictureswithipads.tumblr.com/
![Page 12: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/12.jpg)
para o problema!
http://peopletakingpictureswithipads.tumblr.com/
![Page 13: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/13.jpg)
esforço humano esforço da máquina
![Page 14: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/14.jpg)
F.C.P.P. = dev <3 + machine <3
![Page 15: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/15.jpg)
![Page 16: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/16.jpg)
key-value store
• open source
• in memory (most of the times)
• data-structure server
![Page 17: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/17.jpg)
data types• strings
• lists
• sets
• sorted sets
• hashes
![Page 18: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/18.jpg)
just like that data structures class
![Page 19: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/19.jpg)
except it is COOL!
![Page 20: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/20.jpg)
you can turn this
![Page 21: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/21.jpg)
into this
next next next next
head
tail
![Page 22: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/22.jpg)
or this
next next next
![Page 23: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/23.jpg)
things to keep in mind
• all in memory (but not restricted to)
• no joins or iterating over data (but not restricted to)
• you should be able to rebuild data
![Page 24: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/24.jpg)
why?
• sometimes basic data structures is all you need
• some problems are hard to fit in a relational way
![Page 25: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/25.jpg)
redis @ 8tracks = <3sidekiq
feature toggles
caching
player (moved to couchdb, then to mysql)
autocomplete (moved to SOLR, then to ElasticSearch)
tag browsing!
news feed
listening sessions
rate limiter
![Page 26: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/26.jpg)
awesome documentation
![Page 27: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/27.jpg)
![Page 28: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/28.jpg)
sidekiq
![Page 29: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/29.jpg)
![Page 30: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/30.jpg)
what’s indie?tag
artistplaylist
![Page 31: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/31.jpg)
1 SELECT name !! ! FROM tags !! ! WHERE name !! ! LIKE 'ind%' LIMIT 10;!!2 SELECT name !! ! FROM mixes !! ! WHERE name !! ! LIKE 'ind%' LIMIT 10;!!3 SELECT login !! ! FROM users !! ! WHERE login !! ! LIKE 'ind%' LIMIT 10;!!4 SELECT name !! FROM artists !! WHERE name !! LIKE 'ind%' LIMIT 10;!
![Page 32: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/32.jpg)
you could do
1 (SELECT name FROM tags WHERE name LIKE 'ind%' LIMIT 10)!2 UNION ALL!3 (SELECT name FROM mixes WHERE name LIKE 'ind%' LIMIT 10);!
![Page 33: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/33.jpg)
the problems
• cross-type search
• I don’t want 10 of each, I want the 10 most relevant ones
• can’t select different column counts
![Page 34: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/34.jpg)
not a fair comparison
people don’t use sql databases for search. they use SOLR (or elasticsearch)
![Page 35: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/35.jpg)
sometimes basic data structures!
is all you need
![Page 36: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/36.jpg)
sorted sets
• score
• add, remove, or update log(n)
• elements in order, fast existence test, fast access to elements in the middle
![Page 37: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/37.jpg)
breaking it down
indie =>
probably irrelevant
relevant
i !! ! ! ! ! ! ! in !!
! ! ! ! ! ! ! ind!! ! ! ! ! ! ! indi!! ! ! ! ! ! ! indie
![Page 38: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/38.jpg)
our collection
The indie summer Indiana Industrial
![Page 39: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/39.jpg)
one zset per n-gram
ind [ ][ ][ ]
indi
indie
![Page 40: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/40.jpg)
which is the same asZADD ind, <score>,
ZADD indi, <score>,
ZADD indie, <score>,
![Page 41: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/41.jpg)
searching
1 ind = $r.zevrange('ind', 0, 10)!!
2 indie = $r.zevrange('indie', 0, 10)!
![Page 42: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/42.jpg)
tag browsing
![Page 43: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/43.jpg)
how?
![Page 44: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/44.jpg)
tags mixes
taggings
![Page 45: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/45.jpg)
a single tag1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 WHERE tg.tag_id = 28!5 LIMIT 10;!
chill
![Page 46: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/46.jpg)
two tags1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 INNER JOIN taggings tg2!5 ON tg2.mix_id = m.id!6 WHERE tg.tag_id = 28!7 AND tg2.tag_id = 12!8 LIMIT 10;! chill
acoustic
![Page 47: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/47.jpg)
adding sort1 SELECT m.* FROM mixes m!2 INNER JOIN taggings tg!3 ON tg.mix_id = m.id!4 INNER JOIN taggings tg2!5 ON tg2.mix_id = m.id!6 WHERE tg.tag_id = 28!7 AND tg2.tag_id = 12!8 ORDER BY m.first_published_at!9 LIMIT 10;!
![Page 48: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/48.jpg)
![Page 49: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/49.jpg)
ZINTERSTORE
http://redis.io/commands/zinterstore
![Page 50: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/50.jpg)
zinterstore
Time complexity: O(N*K)+O(M*log(M))
intersects two sets, then store them in a new different set
we’ll get to this
![Page 51: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/51.jpg)
chill
![Page 52: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/52.jpg)
acoustic
![Page 53: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/53.jpg)
chill acoustic
+
![Page 54: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/54.jpg)
chill acoustic
1 sets = ['chill', 'acoustic']!2 cache_key = 'chill:acoustic:cache'!3 $r.zinterstore(cache_key, sets, :weights => [1, 1])!
![Page 55: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/55.jpg)
chill
chill acoustic
popular
1 sets = ['popular', 'chill', 'acoustic']!2 cache_key = 'popular:chill:acoustic:cache'!3 $r.zinterstore(cache_key, sets, :weights => [1, 0, 0])!
![Page 56: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/56.jpg)
back to the BIG O thing
O(N*K)+O(M*log(M))
- N: smallest input - K: number of sorted sets - M: number of elements of the resulting set
![Page 57: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/57.jpg)
![Page 58: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/58.jpg)
cool stuff worth checking out
• pipelining
• lua scripting
• replication + persistence
![Page 59: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/59.jpg)
lessons
• it consumes a LOT of memory (duuuh)
• sentinel for failure recovery
• DO NOT USE “keys” in production
• duplication is tricky
![Page 60: a 8tracks ama o Redis](https://reader034.vdocuments.us/reader034/viewer/2022051411/547e8ec5b37959532b8b54c6/html5/thumbnails/60.jpg)
gracias!
thank you!
obrigado!