clock skew and other annoying realities in distributed systems (donny nadolny, pagerduty) |...
Post on 16-Apr-2017
114 Views
Preview:
TRANSCRIPT
2016−09−08
Clock Skew, and other annoying realities in
distributed systemsDonny Nadolny
donny@pagerduty.com
#CassandraSummit
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS 2016−09−08
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Probably not: • user tracking / metrics • hit counter / impressions • log data
Should I Care?Yes: • incident management (PagerDuty) • financial info / banking / stocks • online store
2016−09−08
Probably not: • user tracking / metrics • hit counter / impressions • log data
Individual data is low impact
Yes: • incident management (PagerDuty) • financial info / banking / stocks • online store
Individual data is high impact
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Should I Care?
9/16/16MAKING PAGERDUTY MORE RELIABLE USING PXC
Introduction to Reads & Writes
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Cluster: 5 nodes • Replication factor: 3 • Consistency: QUORUM
Cassandra Write
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Write
INSERT INTO table1 …
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Write
INSERT INTO table1 …write
foo
write foo
write foo
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Write
INSERT INTO table1 …
value: foo
write fo
o
write foo
write foo
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Write
INSERT INTO table1 …value: foo
value: foo
write fo
o
write foo
write foo
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Write
INSERT INTO table1 …
Success
value: foo
value: foo
write fo
o
write foo
write foo
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Write
INSERT INTO table1 …
Success
value: foo
value: foo
write fo
o
write foo
write foo
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Read
SELECT * FROM table1 WHERE …
value: foo
value: foo
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Read
SELECT * FROM table1 WHERE …
value: foo
value: foo
read
read
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Read
SELECT * FROM table1 WHERE …
value: foo
value: foo
read
read
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Read
SELECT * FROM table1 WHERE …
value: foo
value: foo
read
read
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Read
SELECT * FROM table1 WHERE …
Success, value: foo
value: foo
value: foo
read
read
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Update
UPDATE table1 …value: foo, t=5
value: foo, t=5
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Update
UPDATE table1 …
value: foo, t=5
write ba
r, t=7
write bar, t=7
write bar, t=7
value: foo, t=5
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Cassandra Update
UPDATE table1 …
value: foo, t=5
value: bar, t=7
write ba
r, t=7
write bar, t=7
write bar, t=7
value: foo, t=5 value: bar, t=7
9/16/16MAKING PAGERDUTY MORE RELIABLE USING PXC
Successful Write?
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Bank Example
t=5
savings: 10000, t=5
savings: 10000, t=5
write …
write …
write …
t=2
INSERT INTO balances …savings: 10000, t=5
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Bank Example savings: 10000, t=5
savings: 10000, t=5
t=5
t=2
Success
INSERT INTO balances …savings: 10000, t=5
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Withdraw 8,000 from ATM: • Read current balance: 10,000
Bank Example savings: 10000, t=5
savings: 10000, t=5
read
read
t=6
t=3
savings: 10000, t=5
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Withdraw 8,000 from ATM: • Read current balance: 10,000 • Update to 2,000
Bank Example savings: 10000, t=5 savings: 2000, t=4
write …writ
e …
t=7
t=4
write
sav
ings
: 20
00, t=
4
savings: 10000, t=5 savings: 2000, t=4
s: 10000, t=5 s: 2000, t=4
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Withdraw 8,000 from ATM: • Read current balance: 10,000 • Update to 2,000 • Dispense 8,000 cash
Bank Example
Success
t=7
t=4
savings: 10000, t=5 savings: 2000, t=4
savings: 10000, t=5 savings: 2000, t=4
s: 10000, t=5 s: 2000, t=4
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• A successful write can really fail • Your clocks are not perfectly synchronized • “I’m running NTP, I’m good” - oh really?
Clock Skew
9/16/16MAKING PAGERDUTY MORE RELIABLE USING PXC
Failed Write?
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write?
INSERT INTO stock_trades …
trade 123: buy 100 BRKA
trade 123…
trade 123…
write …
write trade 123 …
write trade 123 …
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write?
INSERT INTO stock_trades …
trade 123: buy 100 BRKA
trade 123…
trade 123…
write …
write trade 123 …
write trade 123 …
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write?
Connection error
trade 123: buy 100 BRKA
trade 123…
trade 123…
write …
write trade 123 …
write trade 123 …
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write?
INSERT INTO stock_trades …
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write?
Connection Error
Write Timeout
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write?
INSERT INTO stock_trades …
trade 245: buy 100 BRKA
trade 245…
trade 245…
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write? trade 245: buy 100 BRKA
trade 245…
trade 245…
hints: tell nodeA trade 123 … tell nodeB trade 123 … tell nodeC trade 123 …
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Failed Write? trade 245: buy 100 BRKA trade 123: buy 100 BRKA
trade 245… trade 123…
trade 245… trade 123…
write …
write trade 123 …
write trade 123 …
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Full repair • Read repair chance • Hinted handoff
Eventual Consistency
9/16/16MAKING PAGERDUTY MORE RELIABLE USING PXC
Multiple Writes aka “I wish I had transactions”
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Rule: minimum $10,000 end of day balance, monthly fee otherwise
Another Bank Example
2016−09−08
• Rule: minimum $10,000 end of day balance, monthly fee otherwise
Balance checker for each user: s = read savings c = read checking if s + c < 10000 mark user for monthly fee
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Another Bank Example
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Rule: minimum $10,000 end of day balance, monthly fee otherwise
Balance checker for each user: s = read savings c = read checking if s + c < 10000 mark user for monthly fee
Another Bank Example
Transfer money amount = … s = read savings c = read checking write_savings(s - amount) write_checking(c + amount)
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Rule: minimum $10,000 end of day balance, monthly fee otherwise
Balance checker for each user: s = read savings c = read checking if s + c < 10000 mark user for monthly fee
Another Bank Example
Transfer money amount = 5000 s = read savings //7000 c = read checking //6000 write_savings(2000) write_checking(13000)
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Rule: minimum $10,000 end of day balance, monthly fee otherwise
Balance checker for each user: s = read savings //2000 c = read checking //6000 if s + c < 10000 //true mark user for monthly fee
Another Bank Example
Transfer money amount = 5000 s = read savings //7000 c = read checking //6000 write_savings(2000) write_checking(11000)
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
1. “Window of vulnerability is small, hope it doesn’t happen” • The client (your application) can crash
2. “Do the writes in reverse order” • Works for balance checker, but allows overdrawing your account
3. “Use a lock!” • The write can propagate out anyway • How long will you hold the lock for a failed write?
Solutions?
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Writes to multiple columns in the same row (when issued at the same time)
• Writes to multiple rows in one table that have the same partition key (when issued at the same time)
Partition key: the primary key of a table, or the first part of the primary key if it is a compound key
Isolation Guarantees in Cassandra
9/16/16MAKING PAGERDUTY MORE RELIABLE USING PXC
Atomic Batches
2016−09−08
https://en.wikipedia.org/wiki/Atomicity_(database_systems)
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Atomicity“An atomic transaction is an indivisible and irreducible series of database operations such that either all occur, or nothing occurs…
the transaction cannot be observed to be in progress by another database client”
2016−09−08
https://en.wikipedia.org/wiki/Atomicity_(database_systems)
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Atomicity“An atomic transaction is an indivisible and irreducible series of database operations such that either all occur, or nothing occurs…
the transaction cannot be observed to be in progress by another database client”
“An example of an atomic transaction is a monetary transfer from bank account A to account B.”
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
Atomic Batch Write
2016−09−08
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Atomic Batch Write
write ba
tch
write batch
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
Atomic Batch Write
write ba
tch
write batch
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
Atomic Batch Write
write ta
ble2
write table1
write table1
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
Atomic Batch Write
Success
write ta
ble2
write table1
write table1
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
Atomic Batch Write
delet
e batc
h
delete batch
2016−09−08
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Atomic Batch Write
write table1
write table1
2016−09−08
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
Atomic Batch Write
Connection error
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
BEGIN BATCH INSERT INTO table1 … INSERT INTO table2 … INSERT INTO table1 … APPLY BATCH;
Atomic Batch Writewrite table2
write table1w
rite
tab
le1
9/16/16MAKING PAGERDUTY MORE RELIABLE USING PXC
Summary
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• No isolation - you can read partial results • … even without any failures
Summary
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• No isolation - you can read partial results • … even without any failures
• Atomic batches aren't really atomic • also, you give up sequential ordering
Summary
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• No isolation - you can read partial results • … even without any failures
• Atomic batches aren't really atomic • also, you give up sequential ordering
• A write can say it failed but really it succeeded • or it didn’t yet, but will hours later
Summary
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• No isolation - you can read partial results • … even without any failures
• Atomic batches aren't really atomic • also, you give up sequential ordering
• A write can say it failed but really it succeeded • or it didn’t yet, but will hours later
• A write can say it succeeded but really it failed • :(
Summary
2016−09−08
Questions? donny@pagerduty.com
2016−09−08CLOCK SKEW AND OTHER ANNOYING REALITIES IN DISTRIBUTED SYSTEMS
• Idempotency - useful overall in distributed systems • Avoid modifying data
• Critical deletes get a new delete column written + row delete • Truly mutable data can be written to a new column (incrementing a
version number in the column name) • Monitor ntp • Distributed locks with ZooKeeper and a sleep(100) before release • Think hard about ordering & partial failure • Test by adding “if (rng < …) exit or sleep” in between various writes
How do you deal with it?
top related