two-phase commit€¦ · one phase commit? how do we get atomicity/linearizability? - need to apply...
TRANSCRIPT
![Page 1: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/1.jpg)
Two-phase commit
![Page 2: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/2.jpg)
Implications of Two Generals
Cannot get agreement in a distributed system to perform some action at the same time.
What if we want to update data stored in multiple locations? In a linearizable fashion?
Perform group of ops at logical instant in time, not physical instant
![Page 3: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/3.jpg)
Setting
Atomic update to data stored in multiple locations
Ex: Multikey update to a sharded key-value store
Ex: Bank transfer
Want: - Atomicity: all or none - Linearizability: consistent with sequential order - No stale reads, no write buffering
For now, let’s ignore availability
![Page 4: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/4.jpg)
One Phase Commit?
Central coordinator decides, tells everyone else
What if some participants can’t do the request?
- Bank account has zero balance
- Bank account doesn’t exist, …
![Page 5: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/5.jpg)
One Phase Commit?
How do we get atomicity/linearizability?
- Need to apply changes at same logical point in time
- Need all other changes to appear before/after
Acquire read/write lock on each location
- If lock is busy, need to wait
For linearizability, need read/write lock on all locations at same time
![Page 6: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/6.jpg)
Two Phase Commit
Central coordinator asks
Participants commit to commit
- Acquire any locks
- In the meantime no other ops allowed on that key
- Delay other concurrent 2PC operations
Central coordinator decides, tells everyone else
- Release locks
![Page 7: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/7.jpg)
Calendar event creation
Doug Woos has three advisors (Tom, Zach, Mike)
Want to schedule a meeting with all of them
- Let’s try Tues at 11, people are usually free then
Calendars all live on different nodes!
Other students also trying to schedule meetings
Nodes can fail, messages can be dropped (of course)
![Page 8: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/8.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
![Page 9: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/9.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Meet at 11 on Tues
![Page 10: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/10.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
OK
![Page 11: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/11.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Meeting Doug @ 11 on Tues
![Page 12: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/12.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Meet at 11 on Tues
Meeting Doug @ 11 on Tues
![Page 13: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/13.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
OK
Meeting Doug @ 11 on Tues
![Page 14: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/14.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Meeting Doug @ 11 on Tues
Meeting Doug @ 11 on Tues
![Page 15: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/15.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Meet at 11 on Tues
Meeting Doug @ 11 on Tues
Meeting Doug @ 11 on Tues
![Page 16: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/16.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Busy!
Meeting Doug @ 11 on Tues
Meeting Doug @ 11 on Tues
![Page 17: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/17.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Meeting Doug @ 11 on Tues
Meeting Doug @ 11 on Tues
![Page 18: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/18.jpg)
Calendar event creation (wrong)
Tom Mike Zach
Doug
Meeting Doug @ 11 on Tues
Meeting Doug @ 11 on Tues
![Page 19: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/19.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
![Page 20: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/20.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Meet at 11 on Tues
![Page 21: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/21.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
OK
![Page 22: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/22.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
![Page 23: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/23.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
Meet at 11 on Tues
![Page 24: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/24.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues OK
![Page 25: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/25.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
Maybe Meeting Doug @ 11 on Tues
![Page 26: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/26.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
Maybe Meeting Doug @ 11 on Tues
Meet at 11 on Tues
![Page 27: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/27.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
Maybe Meeting Doug @ 11 on Tues
Busy!
![Page 28: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/28.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
Maybe Meeting Doug @ 11 on Tues
![Page 29: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/29.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
Maybe Meeting Doug @ 11 on Tues
Never mind!
![Page 30: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/30.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
![Page 31: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/31.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
Maybe Meeting Doug @ 11 on Tues
Never mind!
![Page 32: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/32.jpg)
Calendar event creation (better)
Tom Mike Zach
Doug
![Page 33: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/33.jpg)
Two-phase commit
Atomic commit protocol (ACP)
- Every node arrives at the same decision
- Once a node decides, it never changes
- Transaction committed only if all nodes vote Yes
- In normal operation, if all processes vote Yes the transaction is committed
- If all failures are eventually repaired, the transaction is eventually either committed or aborted
![Page 34: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/34.jpg)
Two-phase commitRoles:
- Participants (Mike, Tom, Zach): nodes that must update data relevant to the transaction
- Coordinator (Doug): node responsible for executing the protocol (might also be a participant)
Messages:
- PREPARE: Can you commit this transaction?
- COMMIT: Commit this transaction
- ABORT: Abort this transaction
![Page 35: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/35.jpg)
2PC without failuresCoordinator Participant Participant
PreparePrepare
Yes
YesCommit
Commit
Yes
![Page 36: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/36.jpg)
2PC without failuresCoordinator Participant Participant
PreparePrepare
Yes
NOABORT
ABORT
Nope
![Page 37: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/37.jpg)
Failures
In the absence of failures, 2PC is pretty simple!
When can interesting failures happen?
- Participant failures?
- Coordinator failures?
- Message drops?
![Page 38: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/38.jpg)
Participant failures: Before sending response?
Coordinator Participant ParticipantPrepare
Prepare
Yes
No Abort
Abort
Decision?
![Page 39: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/39.jpg)
Participant failures: After sending vote?
Coordinator Participant ParticipantPrepare
Prepare
Yes
YesCommit
Commit
Yes
![Page 40: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/40.jpg)
Participant failures: Lost vote?
Coordinator Participant ParticipantPrepare
Prepare
YesYes
No Abort
Abort
Decision?
![Page 41: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/41.jpg)
Coodinator failures: Before sending prepare
Coordinator Participant Participant
PreparePrepare
YesYes
Yes CommitCommit
![Page 42: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/42.jpg)
Coordinator failures: After sending prepare
Coordinator Participant ParticipantPrepare
Prepare
Yes YesYes Commit
Commit
PreparePrepare
![Page 43: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/43.jpg)
Coordinator failures: After receiving votes
Coordinator Participant ParticipantPrepare
Prepare
Yes YesYes Commit
Commit
PreparePrepare
Yes Yes
![Page 44: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/44.jpg)
Coordinator failures: After sending decision
Coordinator Participant ParticipantPrepare
Prepare
YesCommit
Yes Yes
Commit
Decision?
![Page 45: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/45.jpg)
Do we need the coordinator?Coordinator Participant Participant
PreparePrepare
Yes
Commit
Commit
Yes Yes
Decision?
![Page 46: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/46.jpg)
Can the Participants Decide Amongst Themselves?
Coordinator Participant ParticipantPrepare
Prepare
Yes or
No?
Commit?
Decision?
Yes
Yes
![Page 47: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/47.jpg)
Can the Participants Decide Amongst Themselves?
• Yes, if the participants can know for certain that the coordinator has failed
• What if the coordinator is just slow?
• Participants decide to commit!
• Coordinator times out, declares abort!
![Page 48: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/48.jpg)
2PC is a blocking protocol
• A blocking protocol is one that cannot make progress if some of the participants are unavailable (either down or partitioned).
• It has fault-tolerance but not availability.
• This limitation is fundamental.
![Page 49: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/49.jpg)
Can We Make 2PC Non-Blocking?
• Paxos is non-blocking
• We can use Paxos to update individual keys
• Can we use Paxos to update multiple keys?
• If both are on the same shard, easy
• What if on different shards?
![Page 50: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/50.jpg)
State machine
Paxos
Lab 4
State machine
State machine
Paxos
State machine
Paxos
Paxos
Shard master
![Page 51: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/51.jpg)
State machine
Paxos
Lab 4
State machine
State machine
Paxos
State machine
Paxos
Paxos
Shard master
2PC
2PC
Coordinator
![Page 52: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/52.jpg)
State machine
Paxos
Lab 4
State machine
State machine
Paxos
State machine
Paxos
Paxos
Shard master
2PC
2PC
Coordinator
![Page 53: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/53.jpg)
2PC on PaxosCoordinator Participant Participant
PreparePrepare
Yes
YesCommit
Commit
Yes
Paxos
Paxos
Paxos
Paxos
Paxos: state machine replication of operation log
![Page 54: Two-phase commit€¦ · One Phase Commit? How do we get atomicity/linearizability? - Need to apply changes at same logical point in time - Need all other changes to appear before/after](https://reader034.vdocuments.us/reader034/viewer/2022052022/6036b031c39101244d58aafa/html5/thumbnails/54.jpg)
Two Phase Commit on PaxosClient requests multi-key operation at coordinator
Coordinator logs request
- Paxos: available despite node failures
Coordinator sends prepare
Replicas decide to commit/abort, log result
- Paxos: available despite node failures
Coordinator collects replies, log result
- Paxos: available despite node failures
Coordinator sends commit/abort
Replicas record result
- Paxos: available despite node failures