riak on drugs (and the other way...
TRANSCRIPT
![Page 1: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/1.jpg)
Riak on Drugs(and the other way around)
Kresten Krab Thorup, Jørn LarsenTrifork
@drkrab
onsdag den 12. oktober 2011
![Page 2: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/2.jpg)
In this talk...
2
■ About Common Medicine Card
■ Building a Decentralized Architecture
■ Conclusion
onsdag den 12. oktober 2011
![Page 3: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/3.jpg)
A Medicine Card
3
■ For a person
■ List of current drug treatments
■ With prescriptions and related events
onsdag den 12. oktober 2011
![Page 4: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/4.jpg)
Common Medicine Card
4
onsdag den 12. oktober 2011
![Page 5: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/5.jpg)
Common Medicine Card
5
onsdag den 12. oktober 2011
![Page 6: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/6.jpg)
6
15-30 existing systems+150k users
SOAP
onsdag den 12. oktober 2011
![Page 7: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/7.jpg)
7
Master
MySQL
Slave
MySQL
Front Endweb services/app
Front Endweb services/app
Front Endweb services/app
Architecture
onsdag den 12. oktober 2011
![Page 8: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/8.jpg)
Overload
8onsdag den 12. oktober 2011
![Page 9: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/9.jpg)
Database
9
Prescriptions Replica
Treatments+ Events
Front Ends
SOAP endpointWeb applicationBusiness Logic
Prescription Service
Front Ends
SOAP endpointWeb applicationBusiness Logic
onsdag den 12. oktober 2011
![Page 10: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/10.jpg)
Distributed Architecture
10
■ Availability: Run in multiple data centers
■ Scalability: Prepare the system forexpected growth
onsdag den 12. oktober 2011
![Page 11: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/11.jpg)
Riak Data Store
■ Fit the general requirements
■ Availability + Scalability
■ Operational improvements
■ Challenges
■ Key/Value Store, vs Relational Model
■ New technology, many unknowns
11
onsdag den 12. oktober 2011
![Page 12: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/12.jpg)
12
onsdag den 12. oktober 2011
![Page 13: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/13.jpg)
12
onsdag den 12. oktober 2011
![Page 14: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/14.jpg)
12
onsdag den 12. oktober 2011
![Page 15: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/15.jpg)
12
onsdag den 12. oktober 2011
![Page 16: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/16.jpg)
12
onsdag den 12. oktober 2011
![Page 17: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/17.jpg)
12
onsdag den 12. oktober 2011
![Page 18: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/18.jpg)
13
onsdag den 12. oktober 2011
![Page 19: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/19.jpg)
13
onsdag den 12. oktober 2011
![Page 20: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/20.jpg)
13
onsdag den 12. oktober 2011
![Page 21: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/21.jpg)
14
onsdag den 12. oktober 2011
![Page 22: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/22.jpg)
14
coordinate
onsdag den 12. oktober 2011
![Page 23: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/23.jpg)
15
sync
onsdag den 12. oktober 2011
![Page 24: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/24.jpg)
16
scalable and available
system captures write conflicts
resolve lazily (read repair)
onsdag den 12. oktober 2011
![Page 25: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/25.jpg)
Database
17
Prescriptions Replica
Drug Treatments
Front Ends
SOAP endpointWeb applicationBusiness Logic
Prescription Service
Front Ends
SOAP endpointWeb applicationBusiness Logic
onsdag den 12. oktober 2011
![Page 26: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/26.jpg)
MySQL
Master/Slave
18
Front End
SOAP endpointWeb applicationBusiness Logic
Prescription Service
Riak
4 nodes x2 data centers
Front Ends
SOAP endpointWeb applicationBusiness Logic
onsdag den 12. oktober 2011
![Page 27: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/27.jpg)
Challenges
19
■ Data model: To go from Relational model to Key/Value model
■ Experiences with Riak’s backends
■ A true war story
onsdag den 12. oktober 2011
![Page 28: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/28.jpg)
Data Model
■ Integrity without transactions
■ Riak’s default storage keeps all keys in memory
■ Dealing with Write Conflicts
20
onsdag den 12. oktober 2011
![Page 29: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/29.jpg)
First Attempt: Using Links
■ Ran very poorly on Virtual Hardware
■ Trying to figure out how to handle conflicts
21
PersonKey: Person-IDLinks: Prescription-ID
PrescriptionKey: Prescription-IDContent: Protobuf+GZip
~5 million ~200 million
onsdag den 12. oktober 2011
![Page 30: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/30.jpg)
Second Take
■ Very simple: read - resolve - modify - write
■ Integrity: 1 person ⬌ 1 record
■ Performance good: 1 disk access
■ All keys fit in memory22
PrescriptionsKey: Person-IDContent: Medicine Record
~5 million
onsdag den 12. oktober 2011
![Page 31: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/31.jpg)
Read Repair
■ On every read, we handle write conflicts
■ If so, auto-merge[*], store and re-read
■ Resolve: Merging is business logic; some merge actions need user attention, others don’t.
23
onsdag den 12. oktober 2011
![Page 32: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/32.jpg)
Conclusions
■ 1 key per person 500MB in memory
■ Availability, hot replace of HW
■ Continuous Deployment
■ 10x in performance gain and Scalable
■ 2500 TPS with 10% load of nodes
■ Fully deployed 5000 TPS needed
■ Dramatic reduction in cost of HW and Licences24
onsdag den 12. oktober 2011
![Page 33: Riak on Drugs (and the other way around)nosqlroadshow.com/dl/basho-roadshow-london-2011/slides/JrnLarse… · Data model: To go from Relational model to Key/Value model Experiences](https://reader035.vdocuments.us/reader035/viewer/2022070813/5f0cc12d7e708231d436f7ee/html5/thumbnails/33.jpg)
Thank You.
25
@drkrab
onsdag den 12. oktober 2011