patterns for key-value stores
DESCRIPTION
javazone, 2012, redis, riakTRANSCRIPT
![Page 1: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/1.jpg)
PATTERNS FOR KEY-VALUE STORE
JavaZone 2012
Ole-Martin Mørk 13/09/12
![Page 2: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/2.jpg)
OM MEG
Ole-Martin Mørk
Scientist
Bekk Consulting AS
twitter: olemartin
![Page 3: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/3.jpg)
INTRODUCTION
DIFFERENT KV STORES
GENERAL PATTERNS
RIAK PATTERNS
REDIS PATTERNS
RIAK+REDIS PATTERNS
SUMMARY
AGENDA
![Page 4: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/4.jpg)
KEY VALUE session_1 Functional Programming You Already Know
session_2 Fra enkel J2SE til Grid computing med GigaSpaces XAP
session_3 Reinventing software quality
session_4 Taming Java Agents
session_5 Memory Leaks in Java
session_6 Linking data without common identifiers
session_7 Avinstaller Java nå!
MAP
![Page 5: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/5.jpg)
PUT
![Page 6: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/6.jpg)
GET
![Page 7: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/7.jpg)
DEL
![Page 8: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/8.jpg)
KEY VALUE STORE
![Page 9: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/9.jpg)
select value from lookup where key=?
![Page 10: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/10.jpg)
HVORFOR BENYTTE KEY-VALUE STORES? 10
Speed
![Page 11: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/11.jpg)
Distribution
![Page 12: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/12.jpg)
HVORFOR BENYTTE KEY-VALUE STORES? 12
When all you have is a hammer..
![Page 13: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/13.jpg)
PERSON OG ADRESSER
Person
Adresse
Tlf Epost
Twitter Facebook
![Page 14: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/14.jpg)
navn: “Ole-Martin Mørk”,eposter: epost: ”me@ole - martin.net” type: “jobb”, epost: “[email protected]” tlf: type: “mobil”, tlf: “90252338” adresser: type: “hjem” adresse: gate: “Lundliveien 30b”, sted: “Oslo”
type: “jobb” adresse: postboks: “postboks 134, Sentrum”, sted: “Oslo” postnummer: “0150”
![Page 15: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/15.jpg)
{navn: “Ole-Martin Mørk”,eposter: [ { epost: ”me@ole - martin.net” }, { type: “jobb”, epost: “[email protected]” }],tlf: [{ type: “mobil”, tlf: “90252338” }],adresser: [{ type: “hjem”, adresse: { gate: “Lundliveien 30b”, sted: “Oslo” } }, { type: “jobb”, adresse: {postboks: “postboks 134, Sentrum”, sted: “Oslo”, postnummer: “0150” }}]}
![Page 16: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/16.jpg)
![Page 17: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/17.jpg)
HTML5 LOCALSTORAGE
Demo Klikk!
![Page 18: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/18.jpg)
![Page 19: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/19.jpg)
CASSANDRA 19
![Page 20: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/20.jpg)
DETTE ER FINE DATABASER 20
Why?
![Page 21: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/21.jpg)
PATTERNS
![Page 22: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/22.jpg)
Keys
![Page 23: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/23.jpg)
WELL-FORMED KEYS
app:space:what:id
![Page 24: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/24.jpg)
LOOKUPS ON OTHER DATA 24
userId 1234
username olemartin
email [email protected]
twitter olemartin
user:id:1234
user:twitter:olemartin user:id:1234
![Page 25: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/25.jpg)
RELATIONS
userId 1234
username olemartin
email [email protected]
twitter olemartin
user:id:1234
user:id:4321
userId 4321
username grolisbeth
brother user:id:1234
![Page 26: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/26.jpg)
Hash
![Page 27: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/27.jpg)
HASHING
<hash> modulo <antall servere> = serverId
![Page 28: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/28.jpg)
1 2 3 4 3 4 3 2
HASHING
1 1 2
![Page 29: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/29.jpg)
KONSISTENT HASHING
A
BC
![Page 30: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/30.jpg)
KONSISTENT HASHING
A
BC
![Page 31: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/31.jpg)
KONSISTENT HASHING
A
BC
D A
B
C A
B C D
![Page 32: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/32.jpg)
Use cases
![Page 33: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/33.jpg)
EKSEMPLER PÅ APPLIKASJONER SOM KAN KLE K/V
RDBMS + KEY/VALUE
![Page 34: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/34.jpg)
![Page 35: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/35.jpg)
![Page 36: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/36.jpg)
![Page 37: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/37.jpg)
RIAK
Bucket
Bucket
Bucket
Key
Key
Key
Key
![Page 38: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/38.jpg)
C A P
onsistency vailability artition
![Page 39: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/39.jpg)
N W R
![Page 40: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/40.jpg)
N > W
![Page 41: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/41.jpg)
N > R
![Page 42: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/42.jpg)
![Page 43: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/43.jpg)
PATTERNS
![Page 44: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/44.jpg)
RIAK OG MAP REDUCE
![Page 45: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/45.jpg)
MAP REDUCE
shopping:13092012:userId
![Page 46: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/46.jpg)
SECONDARY INDEXES 46
Secondary indexes
![Page 47: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/47.jpg)
SECONDARY INDEXES 47
userId 1234
username olemartin
email [email protected]
twitter olemartin
user:id:1234
user:twitter:olemartin user:id:1234
twitter:id_bin olemartin
email:id_bin [email protected]
user:sex_bin Male
user:age_int 35
![Page 48: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/48.jpg)
Pre Commit
Post Commit
![Page 49: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/49.jpg)
![Page 50: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/50.jpg)
Hash
Lists
Sets
Sorted Sets
Strings
![Page 51: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/51.jpg)
PATTERNS
![Page 52: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/52.jpg)
SELECT * FROM foo WHERE ... ORDER BY time DESC LIMIT 10
![Page 53: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/53.jpg)
PATTERNS FOR MILJØER MED RDBMS
RDBMS REDIS
![Page 54: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/54.jpg)
![Page 55: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/55.jpg)
STATISTIKK
Antall sider lest av bruker <pages:userId, count>
incr pages:userId
Mest leste i dag: <most_read:300-2012, <url, count>,<url, count>>
zrangebyscore most_read:300-2012 -inf +inf
Unike besøkende: <page:url:300-2012, <userId>, <userId>, <userId>>
scard page:url:300-2012
![Page 56: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/56.jpg)
SERIALIZATION
0 ms 10000 ms 20000 ms 30000 ms 40000 ms 50000 ms 60000 ms 70000 ms 80000 ms 90000 ms
100000 ms
![Page 57: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/57.jpg)
KRYO IN REDIS
<bean id="redisTemplate" class="org.springframework.data.redis.core.RedisTemplate"> <property name="connectionFactory" ref="connectionFactory"/> <property name="defaultSerializer" ref="serializer"/></bean><bean id="serializer" class="no.bekk.redis.DiggerSerializer"/>
![Page 58: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/58.jpg)
KRYO
public class DiggerSerializer<T> implements RedisSerializer<T> {
private Kryo kryo = new Kryo();
public DiggerSerializer() { kryo.register(Story.class); } public byte[] serialize(T t) { return new ObjectBuffer(kryo).writeClassAndObject(t); } public T deserialize(byte[] bytes) { return (T) new ObjectBuffer(kryo).readClassAndObject(bytes); }}
![Page 59: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/59.jpg)
REDDIT KLONE
1. story:id – holder styr på id-en til en story
2. story:id, value – holder selve story-objektet
3. story:stories_zset – holder styr på antall stemmer
4. story:tag:storyId – holder styr på tags knyttet til en story
5. tag:tagId – holder styr på stories knyttet til en tag
![Page 60: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/60.jpg)
REDIS OG RIAK
+
=
![Page 61: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/61.jpg)
![Page 62: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/62.jpg)
Produktkatalog
Product catalog
![Page 63: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/63.jpg)
RIAK
Products
product:2
product:1
![Page 64: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/64.jpg)
POST COMMIT
Products
Redis
RSS
![Page 65: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/65.jpg)
REDIS
catalog:product:id binary data
catalog:product:today:sold_zset <productId, counter>
catalog:product:week:sold_zset <productId, counter>
catalog:product:year:sold_zset <productId, counter>
![Page 66: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/66.jpg)
RESULT
![Page 67: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/67.jpg)
Summary
![Page 68: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/68.jpg)
SUMMARY
1. KV-stores kommer i alle farger og fasonger
2. Gode nøkkelnavn
3. Bruk god tid på datamodellering
4. Konsistent hashing
5. Bruk KV-store sammen med RDBMS
6. Bruk KV-store for domener som matcher
7. Bruk databasen til det den er god til 8. Kombiner gjerne flere type KV-stores
9. La kreativiteten blomstre
![Page 69: Patterns for key-value stores](https://reader034.vdocuments.us/reader034/viewer/2022051609/547ea2ccb4af9fd3158b573d/html5/thumbnails/69.jpg)
69
Questions?