Eventual Consistency != Hopeful Consistency

Embracing Optimistic Design in the Persistence Layer

#Cassandra13

Who am I?Christos KalantzisNetflix Inc.Manager – Cloud Persistence Engineering

@chriskalanckalantzis@netflix.comwww.linkedin.com/in/christoskalantzis

#Cassandra13

C* Replication & Consistency Recap• C* is eventually Consistent– That means it WILL get there…eventually

• Eventually is not:– A day from now– A minute from now– A second from now

• In most cases it is milliseconds

#Cassandra13

• C* has tunable consistency– All

– Quorum

– One

C* Replication & Consistency Recap

#Cassandra13

Remember When...?• In early 2000s, Read was scaled out by adding

replication Slaves to a Master DB.• Writes went to a single Master, Reads went to

Slaves

#Cassandra13

Remember When…?• Slaves could lose transactions

#Cassandra13

Remember When…?• No “Repair” function

#Cassandra13

Remember When…?• We trusted it• We still trust it

#Cassandra13

C* Consistency Concerns• “I want high consistency in my Reads/Writes just like I had

in my RDBMS setup”– You never really had it.

• “I want my DB to catch integrity issues”– We’ve been turning FK off for years!

• See Rails, Grails & other MVC frameworks

• Can I trust that C* will replicate my data when writing at CL 1”

#Cassandra13

Netflix Experiment• Created a multi-datacenter C* 1.1.7 cluster of 48 nodes in each

DC• Put load on C* Cluster

– 100K total operations per second (50 K in each DC)• Wrote 1,000,000 records at CL1 in one DC• Read same 1,000,000 records at CL1 in other DC• ALL records were read successfully• You can trust it!

#Cassandra13

Netflix Experiment

#Cassandra13

Optimistic vs. Pessimistic Design• Pessimistic Design– Design with high consistency, you punish your

users 99.9% of the time• Higher consistency = higher latency• Diminished user experience

#Cassandra13

Optimistic vs. Pessimistic Design• Optimistic Design– Trust your data store

• Know your business and your application– Always ask yourself, is it really that important?

• Handle edge cases through contingency plans

#Cassandra13

Low Consistency Example 1Amazon• Inventory system sometimes sells items not

available• Cancel the order• They offer credit of 10% towards future

purchase#Cassandra13

Low Consistency Example 2Banks• The most eventual consistent system• Write a check, which may or may not be covered• Bank will “bounce” the check• Recoup funds if possible and charge a handsome

fee

#Cassandra13

Hurdles Faced• Engineers are stubborn!

– 1+1=2 .. not eventually 2• Middle management is scared

– It’s a feat you convinced them to use C*– Now you got to convince them to accept low consistency?

• Engaging Product team to implement some type of contingency plans

#Cassandra13

How to Overcome those Hurdles• Prove it through a POC• Show them the benefits of an improved user

experience– Its all about the user

• You may be working for the wrong company– We’re hiring!– jobs.netflix.com

#Cassandra13

Q&A

#Cassandra13

Thank you!

#Cassandra13