Re-Configurable Byzantine Quorum System

Lei KongS. Arun

Mustaque AhamadDoug Blough

Related Work

Byzantine Quorum System – D. Malkhi and M. Reiter: Increase read/write quorum intersection size to tolerate arbitrary server failures.

Dynamic Byzantine Quorum System – L. Alvisi, D. Malkhi, E. Pierce M. Reiter and R. N. Wright: define fault resilience threshold b as an variable, so it can be adjusted dynamically.

Fault Detection for Byzantine Quorum Systems – Alvisi, Malkhi, Pierce and Reiter, IEEE Trans. on Parallel and Distributed Systems, Sept. 2001.

Our Contribution

Explicitly add fault detection mechanism into the system, and remove faulty servers for smaller quorum size and lighter system load.

Proxy scheme enables servers to monitor each other instead of using clients to monitor servers.

A new statistical fault detection technique.

System Model

Server failure model is Byzantine failure, the number of concurrent failures is assumed to be in the range [bmin,bmax].

The data service protocol is based on threshold masking quorum system, and but our re-configurable approach could be applied to other types of quorum systems.

Quorum data operations are done through proxy servers. Communication channels are assumed to be asynchronous

but reliable.

System Architecture

Server faultdetection

Clients

Distributed store

PP

Quorum Variables

Four quorum variables N, B, Qmin and S are defined.

Defining system size a variable makes it possible to remove faulty nodes out of the system.

Suppose Q(V) stands for the number servers that are currently in the system and belong to the quorum of the most recently finished write of V, then Qmin is the minimum value of Q(V) for all data objects in the system.

S is an boolean array used to indicate server status. Quorum variable operations use a static quorum setting.

Read/Write Quorum Variables

Read Protocol– Read from a quorum of 3bmax+1 servers

– Return the value that is returned by at least bmax+1 servers and is not countermanded, return error if such a pair doesn’t exist.

Write Protocol– Write to a quorum of n- bmax servers

Write quorum size = n-bmax, read quorum size = 3bmax+1,

then the intersection size is at least 2bmax+1.

Read Data Object V

1. The client executes a read on quorum variables;

2. The client randomly chooses a server as the proxy and choose a read quorum of size n+2b+1-qmin;

3. The client sends the read request and the chosen quorum to the proxy;

4. The proxy first read from n+b+bmin+1-qmin servers in the chosen quorum and forwards results to the client;

Read Data Object V - continue

5. Among all pairs returned by at least bmin+1 servers, if the one with the highest timestamp doesn’t have at least b+1 representatives, the proxy read from the rest of the servers in the read quorum and forwards results to the client.

6. The client chooses pairs returned by at least bmin+1 servers, if the one with the highest timestamp has at least b+1 representatives, then return it, otherwise restart from step 2.

Write Data Object V

1. The client executes a read on V to get the current timestamp of V and current quorum variable values;

2. If read quorum size increases according to quorum variable values received in step one, then read quorum variable values from all server nodes, until at least n-bmax servers return the same values;

3. The client generate a new timestamp for V;

4. The client chooses its proxy and write quorum of size [(n+2b+1)/2];

Write Data Object V - continue

5. The client sends the write request and the write quorum to the proxy.

6. The proxy writes to the servers in the write quorum and forwards back server confirmations back to the client;

7. The client check server confirmations, restart from step 2 if error detected.

Quorum Sizes for Data Objects

Write quorum size for data objects: [(n+2b+1)/2] Read quorum size for data objects: n+2b+1-qmin, and qmin <=

any write quorum size, then the intersection size is at least 2b+1.

Misc.

Message authentication code is used to protect message integrity, faulty proxy servers could only drop messages, they cannot tamper with them.

Proxy nodes do not forward client read request MACs to server nodes, which makes it feasible for server nodes to use explicit testing on each other.

Reducing the overhead of reading quorum variables: cached them on client side.

Simulation - Plot of n vs. b

Simulation - Comparison of Read Quorum Size

Simulation - Comparison of Write Quorum Size

Simulation -Comparison of Workload (%r.|Qr|+%w.|Qw|)/n

Fault Detection and Diagnosis

Identify faulty servers thereby enabling their removal Fault detection done by monitoring a server’s responses to

read requests over time Fault detection probability close to 1 for a wide range of pic

and very low false alarm probability To avoid detection, a faulty server must operate as though it

were a correct server

Fault Detection Algorithm

Observe #correct responses over several read operations returned by each server

Two-tiered diagnosis – proxy-node level and diagnosis-node level

Faulty servers try to avoid getting detected– If a faulty server has the correct value for a variable, it returns an

incorrect value with probability “pic”• pic=1 corresponds to a server blatantly exposing itself as faulty• pic=0 corresponds to a correct server

Analysis based on Hypothesis Testing

Probability a correct server returns “u” correct responses in “r” read operations is rCu (|Qw|/n)u (1-|Qw|/n)r-u , where |Qw| is the size of the write quorum and n is the total number of servers in the system.

Plot for n=50, |Qw|=34, r=10000

Plot for n=50, |Qw|=34, r=1000

Plot for n=50, |Qw|=34, r=100

Faulty Server Modeling and Analysis

If 0.05 is the false alarm probability tolerated at the proxy-node level, then let uth be the maximum value of u such that Σu

i=0 rCi (|Qw|/n)i (1-|Qw|/n)r-i ≤ 0.05. A server is diagnosed as faulty if it returns uth or fewer

correct responses in “r” read operations. Let pic denote the probability with which a faulty server

returns an incorrect response when it has the correct value for the data object being read.

Probability that a faulty server will be detected in “r” read operations is Σuth

i=0 rCi ((1-pic)|Qw|/n)i (1-|Qw|/n+pic|Qw|/n)r-i

Faulty Server Detection Prob. for n=50,|Qw|=34

Prob. Faulty Server is Undetected, n=50, |Qw|=34

Faulty Server Det. Prob. for different |Qw|, n=50

Fault Detection Algorithm at the Diagnosis Node

Proxy nodes report detected state of servers (faulty or not-faulty) to the diagnosis node every “r” read operations

If at any time a server has been found to be faulty by “m” servers, then that server is diagnosed as faulty and removed from the system

Choice of “m” depends on the desired final false alarm probability

Diagnosis-Node Fault Detection Algorithm (cont’d)

If the final desired false alarm probability is 10-4, then let m2 be the smallest m1 such that n-bmaxCm1 0.05m1 (1-0.05)n-bmax-m1 ≤ 10-4 , where 0.05 is the false alarm probability at the proxy-node level.

Then “m” is given by m2+bmax.

An assumption: The behavior of faulty servers is independent of the identity of the proxy servers.

More Algorithm Specifications and Features

Data servers keep track of the |Qw|/n ratio for each object they store and return this value along with the data object during a read.

Several values of “r” considered to increase chances of detecting faulty servers at the earliest.

Statistical analysis over several read operations tolerates low levels of concurrency between reads and writes.

Above analysis holds only if write quorums are chosen randomly. Analysis can be easily modified to suit other quorum-picking strategies.

Future Work

Make use of background dissemination, explore possible approaches to adjust Qmin: termination determination, objects groups.

Allow new nodes to be added into the system, add replacements for removed faulty servers.