webinar: eventual consistency != hopeful consistency

Eventual Consistency != Hopeful Consistency

Embracing Optimistic Design in the Persistence Layer

Who am I?Christos KalantzisNetflix Inc.Engineering Manager – Cloud Database Engineering

@chriskalanckalantzis@netflix.comwww.linkedin.com/in/christoskalantzis

C* Replication & Consistency Recap• C* is eventually Consistent– That means it WILL get there…eventually

• Eventually is not:– A day from now– A minute from now– A second from now

• In most cases it is milliseconds

• C* has tunable consistency– All

– Quorum

– One

C* Replication & Consistency Recap

Remember When...?• In early 2000s, Read was scaled out by adding

replication Slaves to a Master DB.• Writes went to a single Master, Reads went to

Slaves

Remember When…?• Slaves could lose transactions

Remember When…?• No “Repair” function

Remember When…?• We trusted it• We still trust it

C* Consistency Concerns• “I want high consistency in my Reads/Writes just like I had

in my RDBMS setup”– You never really had it.

• “I want my DB to catch integrity issues”– We’ve been turning FK off for years!

• See Rails, Grails & other MVC frameworks

• Can I trust that C* will replicate my data when writing at CL 1”

Netflix Experiment• Created a multi-datacenter C* 1.1.7 cluster of 48 nodes in each

DC• Put load on C* Cluster

– 100K total operations per second (50 K in each DC)• Wrote 1,000,000 records at CL1 in one DC• Read same 1,000,000 records at CL1 in other DC• ALL records were read successfully• You can trust it!

Netflix Experiment

Optimistic vs. Pessimistic Design• Pessimistic Design– Design with high consistency, you punish your

users 99.9% of the time• Higher consistency = higher latency• Diminished user experience

Optimistic vs. Pessimistic Design• Optimistic Design– Trust your data store

• Know your business and your application– Always ask yourself, is it really that important?

• Handle edge cases through contingency plans

Low Consistency Example 1Retail

Low Consistency Example 2Banking

Hurdles Faced• Engineers are stubborn!

– 1+1=2 .. not eventually 2• Middle management is scared

– It’s a feat you convinced them to use C*– Now you got to convince them to accept low consistency?

• Engaging Product team to implement some type of contingency plans

How to Overcome those Hurdles• Prove it through a POC• Show them the benefits of an improved user

experience– Its all about the user

• You may be working for the wrong company– We’re hiring!– jobs.netflix.com

Q&A

Thank you!