lock service with paxos in erlang

Post on 24-May-2015

776 Views

Category:

Technology

1 Downloads

Preview:

Click to see full reader

DESCRIPTION

Overview of my implementation of a wannabe highly available lock service over Paxos algorithm in Erlang

TRANSCRIPT

. . . . . .

MotivationImplementation

EvaluationConclusions

.

.

. ..

.

.

Lock Service with Paxos

Presenter: Emmanouil DimogerontakisTeammate: Muhammet OrazovSupervisor: Johan Montelius

CC© BY:© C©

EMDC KTH

December 5, 2012

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 1/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. .1 Motivation

.. .2 Implementation

ArchitectureFault Tolerance & Optimizations

.. .3 Evaluation

Central LockExperiments

.. .4 Conclusions

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 2/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. Paxos

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 3/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. Paxos

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 3/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. From Paxos to a Lock Service

Not Trivial

Paxos Replicated State Machine

Slot queue → Sequence of lock holders

Paxos round number = Slot number

Paxos decides who gets each slot

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 4/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. From Paxos to a Lock Service

Not Trivial

Paxos Replicated State Machine

Slot queue → Sequence of lock holders

Paxos round number = Slot number

Paxos decides who gets each slot

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 4/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. From Paxos to a Lock Service

Not Trivial

Paxos Replicated State Machine

Slot queue → Sequence of lock holders

Paxos round number = Slot number

Paxos decides who gets each slot

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 4/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. From Paxos to a Lock Service

Not Trivial

Paxos Replicated State Machine

Slot queue → Sequence of lock holders

Paxos round number = Slot number

Paxos decides who gets each slot

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 4/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. From Paxos to a Lock Service

Not Trivial

Paxos Replicated State Machine

Slot queue → Sequence of lock holders

Paxos round number = Slot number

Paxos decides who gets each slot

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 4/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. .1 Motivation

.. .2 Implementation

ArchitectureFault Tolerance & Optimizations

.. .3 Evaluation

Central LockExperiments

.. .4 Conclusions

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 5/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Abstract View

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 6/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Detailed View

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 7/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Use Case

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 8/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Process Fault Tolerance

Supervisor Tree

Process restarts aware that it has crashed

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 9/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Paxos Fault Tolerance

Acceptor persists asynchronously its state

Learner persists asynchronously the slot queue

Assignor persists asynchronously its state

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 10/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Paxos Fault Tolerance

Acceptor persists asynchronously its state

Learner persists asynchronously the slot queue

Assignor persists asynchronously its state

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 10/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Paxos Fault Tolerance

Acceptor persists asynchronously its state

Learner persists asynchronously the slot queue

Assignor persists asynchronously its state

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 10/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Optimizations

@COMPLETED:

3 Receiver handles the huge amount of requests

3 Shared Queue between Receiver and Proposer

3 Learner and Acceptor spawn separate writing process

@TODO:

7 Propose lock requests in batches

7 Leader

7 Garbage Collection

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 11/19

. . . . . .

MotivationImplementation

EvaluationConclusions

ArchitectureFault Tolerance & Optimizations

.. Optimizations

@COMPLETED:

3 Receiver handles the huge amount of requests

3 Shared Queue between Receiver and Proposer

3 Learner and Acceptor spawn separate writing process

@TODO:

7 Propose lock requests in batches

7 Leader

7 Garbage Collection

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 11/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. .1 Motivation

.. .2 Implementation

ArchitectureFault Tolerance & Optimizations

.. .3 Evaluation

Central LockExperiments

.. .4 Conclusions

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 12/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. Throughput

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 13/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. Latency

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 14/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. Cost of Availability

# of Paxos Nodes VS Latency and/or Throughput

Will look like:

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 15/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. Cost of Availability

# of Paxos Nodes VS Latency and/or ThroughputWill look like:

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 15/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. Cost of Fault Tolerance

Latency of Node Recovery VS Latency and/or Throughput

Will look like:

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 16/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. Cost of Fault Tolerance

Latency of Node Recovery VS Latency and/or ThroughputWill look like:

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 16/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Central LockExperiments

.. More

Proposer Timeouts VS Latency and/or Throughput

Paxos +/-Persistence VS Latency and/or Throughput

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 17/19

. . . . . .

MotivationImplementation

EvaluationConclusions

.. .1 Motivation

.. .2 Implementation

ArchitectureFault Tolerance & Optimizations

.. .3 Evaluation

Central LockExperiments

.. .4 Conclusions

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 18/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Find golden ratio: Performance vs Availability

Find golden ratio: Performance vs Fault Tolerance

Their combination?

Questions ?

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 19/19

. . . . . .

MotivationImplementation

EvaluationConclusions

Find golden ratio: Performance vs Availability

Find golden ratio: Performance vs Fault Tolerance

Their combination?

Questions ?

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 19/19

. . . . . .

Part I.

.

. ..

.

.

Appendix

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 20/19

. . . . . .

.. Benchmarking Tool

Basho bench

Ability to write custom clients

Can simulate network dispatcher process

Presenter: Emmanouil Dimogerontakis Teammate: Muhammet Orazov Supervisor: Johan Montelius CC© BY:© C©Lock Service with Paxos 1/1

top related