clock skew and other annoying realities in distributed systems (donny nadolny, pagerduty) |...

2016−09−08

Clock Skew, and other annoying realities in

distributed systemsDonny Nadolny

[email protected]

#CassandraSummit

mailto:[email protected]?subject=

CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS 2016−09−08

2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS

Probably not: • user tracking / metrics • hit counter / impressions • log data

Should I Care?Yes: • incident management (PagerDuty) • financial info / banking / stocks • online store

2016−09−08

Probably not: • user tracking / metrics • hit counter / impressions • log data

Individual data is low impact

Yes: • incident management (PagerDuty) • financial info / banking / stocks • online store

Individual data is high impact

CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS

Should I Care?

9/16/16MAKING PAGERDUTY MORE RELIABLE USING PXC

Introduction to Reads & Writes


• Cluster: 5 nodes • Replication factor: 3 • Consistency: QUORUM

Cassandra Write


Cassandra Write

INSERT INTO table1 …


Cassandra Write

INSERT INTO table1 …write

foo

write foo

write foo


Cassandra Write


value: foo

write fo

o

write foo

write foo


Cassandra Write

INSERT INTO table1 …value: foo

value: foo

write fo

o

write foo

write foo


Cassandra Write


Success

value: foo

value: foo

write fo

o

write foo

write foo


Cassandra Read

SELECT * FROM table1 WHERE …

value: foo

value: foo


Cassandra Read


value: foo

value: foo

read

read


Cassandra Read


Success, value: foo

value: foo

value: foo

read

read


Cassandra Update

UPDATE table1 …value: foo, t=5

value: foo, t=5


Cassandra Update

UPDATE table1 …

value: foo, t=5

write ba

r, t=7

write bar, t=7

write bar, t=7

value: foo, t=5


Cassandra Update

UPDATE table1 …

value: foo, t=5

value: bar, t=7

write ba

r, t=7

write bar, t=7

write bar, t=7

value: foo, t=5 value: bar, t=7


Successful Write?


Bank Example

t=5

savings: 10000, t=5

savings: 10000, t=5

write …

write …

write …

t=2

INSERT INTO balances …savings: 10000, t=5


Bank Example savings: 10000, t=5

savings: 10000, t=5

t=5

t=2

Success

INSERT INTO balances …savings: 10000, t=5


• Withdraw 8,000 from ATM: • Read current balance: 10,000

Bank Example savings: 10000, t=5

savings: 10000, t=5

read

read

t=6

t=3

savings: 10000, t=5


• Withdraw 8,000 from ATM: • Read current balance: 10,000 • Update to 2,000

Bank Example savings: 10000, t=5 savings: 2000, t=4

write …writ

e …

t=7

t=4

write

sav

ings

: 20

00, t=

4

savings: 10000, t=5 savings: 2000, t=4

s: 10000, t=5 s: 2000, t=4


• Withdraw 8,000 from ATM: • Read current balance: 10,000 • Update to 2,000 • Dispense 8,000 cash

Bank Example

Success

t=7

t=4



s: 10000, t=5 s: 2000, t=4


• A successful write can really fail • Your clocks are not perfectly synchronized • “I’m running NTP, I’m good” - oh really?

Clock Skew


Failed Write?


Failed Write?

INSERT INTO stock_trades …

trade 123: buy 100 BRKA

trade 123…

trade 123…

write …

write trade 123 …

write trade 123 …


Failed Write?

Connection error


trade 123…

trade 123…

write …

write trade 123 …

write trade 123 …


Failed Write?



Failed Write?

Connection Error

Write Timeout


Failed Write?



trade 245…

trade 245…


Failed Write? trade 245: buy 100 BRKA

trade 245…

trade 245…

hints: tell nodeA trade 123 … tell nodeB trade 123 … tell nodeC trade 123 …


Failed Write? trade 245: buy 100 BRKA trade 123: buy 100 BRKA

trade 245… trade 123…

trade 245… trade 123…

write …

write trade 123 …

write trade 123 …


• Full repair • Read repair chance • Hinted handoff

Eventual Consistency


Multiple Writes aka “I wish I had transactions”


• Rule: minimum $10,000 end of day balance, monthly fee otherwise

Another Bank Example

2016−09−08


Balance checker for each user: s = read savings c = read checking if s + c < 10000 mark user for monthly fee







Transfer money amount = … s = read savings c = read checking write_savings(s - amount) write_checking(c + amount)





Transfer money amount = 5000 s = read savings //7000 c = read checking //6000 write_savings(2000) write_checking(13000)



Balance checker for each user: s = read savings //2000 c = read checking //6000 if s + c < 10000 //true mark user for monthly fee


Transfer money amount = 5000 s = read savings //7000 c = read checking //6000 write_savings(2000) write_checking(11000)


1. “Window of vulnerability is small, hope it doesn’t happen” • The client (your application) can crash

2. “Do the writes in reverse order” • Works for balance checker, but allows overdrawing your account

3. “Use a lock!” • The write can propagate out anyway • How long will you hold the lock for a failed write?

Solutions?


• Writes to multiple columns in the same row (when issued at the same time)

• Writes to multiple rows in one table that have the same partition key (when issued at the same time)

Partition key: the primary key of a table, or the first part of the primary key if it is a compound key

Isolation Guarantees in Cassandra


Atomic Batches

2016−09−08

https://en.wikipedia.org/wiki/Atomicity_(database_systems)


Atomicity“An atomic transaction is an indivisible and irreducible series of database operations such that either all occur, or nothing occurs…

the transaction cannot be observed to be in progress by another database client”

2016−09−08

https://en.wikipedia.org/wiki/Atomicity_(database_systems)


Atomicity“An atomic transaction is an indivisible and irreducible series of database operations such that either all occur, or nothing occurs…

the transaction cannot be observed to be in progress by another database client”

“An example of an atomic transaction is a monetary transfer from bank account A to account B.”


BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;

Atomic Batch Write

2016−09−08



Atomic Batch Write

write ba

tch

write batch



Atomic Batch Write

write ba

tch

write batch



Atomic Batch Write

write ta

ble2

write table1

write table1



Atomic Batch Write

Success

write ta

ble2

write table1

write table1



Atomic Batch Write

delet

e batc

h

delete batch

2016−09−08



Atomic Batch Write

write table1

write table1

2016−09−08



Atomic Batch Write

Connection error



Atomic Batch Writewrite table2

write table1w

rite

tab

le1


Summary


• No isolation - you can read partial results • … even without any failures

Summary



• Atomic batches aren't really atomic • also, you give up sequential ordering

Summary




• A write can say it failed but really it succeeded • or it didn’t yet, but will hours later

Summary




• A write can say it failed but really it succeeded • or it didn’t yet, but will hours later

• A write can say it succeeded but really it failed • :(

Summary

2016−09−08

Questions? [email protected]


• Idempotency - useful overall in distributed systems • Avoid modifying data

• Critical deletes get a new delete column written + row delete • Truly mutable data can be written to a new column (incrementing a

version number in the column name) • Monitor ntp • Distributed locks with ZooKeeper and a sleep(100) before release • Think hard about ordering & partial failure • Test by adding “if (rng < …) exit or sleep” in between various writes

How do you deal with it?

clock skew and other annoying realities in distributed systems (donny nadolny, pagerduty) |...

Software