Gossip and its applicationGossip and its application

Presented by Anna Kaplun

AgendaAgendaTechnical preliminariesGossip algorithms

◦Randomized unbalanced gossip◦unbalanced gossip

ConsensusDistributed computing

Technical preliminariesTechnical preliminariesThe system is synchronousThere are n processors, each with

unique integer name in interval {1,..,n}. n is known to all processors.

Each processor can send a message to any subset of processors in one round. The size of the message assumed to be sufficiently large to carry a complete local state.

Technical preliminaries – Technical preliminaries – Performance metricsPerformance metricsCommunication denotes the total

number of point-to-point messages sentTime is measured as number of rounds.

The round is such number of clock cycles that is sufficient to complete:◦receive messages delivered in previous

round◦Local computations◦Send messages to arbitrary set of

processors and deliver them.

GossipGossipIn the beginning each processor

has an input value called rumor.The goal: every non faulty

processor p◦ knows the rumor of any other

processor q OR

◦ p knows that q has crushed

Randomized unbalanced Randomized unbalanced gossipgossip Processors are partitioned into m groups of balanced size.

m=min{n,2t}◦n – number of processors◦t – maximum number of faulty

processorsEvery group has a leaderOnly leaders send messages while regular nodes may only answer leaders’ requests

Randomized unbalanced Randomized unbalanced gossipgossip Processors are partitioned into w chunks of balanced size.

w is:◦if 2t < n than w=2t◦else w=n-t

Note: If 2t<n then chunks and groups are

the same

Communication graphCommunication graphEvery node is connected to

appropriate leaderLeaders form a graph( )lG m

Is a graph which consists of m nodes and has the

following properties:For each subgraph of size at least m-t ,there is a subgraph P(R) such that:

Is a graph which consists of m nodes and has the

following properties:For each subgraph of size at least m-t ,there is a subgraph P(R) such that:

( )lG m

R G

1 2 1 2

1: P(R)

2 : ( ) / 7

3:The diameter of P(R) is at most 2+30ln(m)

4:If R then P(R ) P(R )

R

P R R

R

its degree is

its degree is

27/22log 74m

Om t

Communication graph Communication graph (m=2t<n)(m=2t<n)

m groupsm leadersconnect nodes

to leadersLeaders form ,

graph( )lG mChunks and groups

are the same

Communication graph Communication graph (m=n≤2t)(m=n≤2t)

m groupsm leadersconnect nodes

to leadersLeaders form ,

graph( )lG m

n-t chunks

Randomized unbalanced Randomized unbalanced gossip -gossip -

local viewlocal viewRumors – all known rumors

initialized: Rumorsp[p]=myRumor

Active – list of crashed processors

initialized: Activep[q]=nill (for every q)

Pending – list of fully informed processors

initialized: Pendingp[q]=nill (for every q)

Randomized unbalanced Randomized unbalanced gossip -gossip -

messagesmessages – carrying the whole local state,

sent along communication graph – requests a local state from

specific node – carrying the whole local

state, sent when the sender knows all rumors (or knows that a certain processor crushed)

– carrying the whole local state, sent as reply to inquiry message.

graph

inquiry

notification

reply

Randomized unbalanced Randomized unbalanced gossip –gossip –the algorithmthe algorithmOnly leaders send messages,

regular nodes only answer queries

Leader starts as collector, when it knows about all nodes it becomes disseminator

Algorithm consists of phases:1. Regular phase – executed T times2. Ending phase – executed 4 times

a. Update the local arrays

b. If p is a collector that has already heard about all nodes than become disseminator

For each processor q:

a. If q is active and q is my neighbor in communication graph than send a graph message to q

b. If I’m collector and q is in the first chunk with a processor about which I haven't heard yet, then send inquiring message

c. I I’m disseminator and q is in the first chunk with a processor that need to be notified, then send notifying message.

d. If q is a collector from which an inquiry message was received then send a reply message to q

Randomized unbalanced gossip – the Randomized unbalanced gossip – the algorithmalgorithm

(regular phase ,executed T times )(regular phase ,executed T times )

graph

inquiry

notification

reply

Chunks are ordered

according to permutation πp

Randomized unbalanced Randomized unbalanced gossip -gossip -the algorithm (regular the algorithm (regular phase)phase) For some leader

pSend graph messages

Take first unknown chunk

from πp, send query

Answer queries

Randomized unbalanced Randomized unbalanced gossip -gossip -the algorithm (regular the algorithm (regular phase)phase)

For some leader p

That collected all rumors

Send graph messages

Take first uninformed chunk

from πp, send notification

Answer queries

for 4 timesa. Update the local arrays

b. If p is a collector that has already heard about all nodes than become disseminator

For each processor q:

a. If I’m collector and I don’t know about q , then send inquiring message

b. I I’m disseminator and q need to be notified, then send notifying message.

c. If q is a collector from which an inquiry message was received then send a reply message to q

Randomized unbalanced Randomized unbalanced gossip -gossip -

the algorithm (ending the algorithm (ending phase)phase)

inquiry

notification

reply

Randomized unbalanced Randomized unbalanced gossip –gossip –updating listsupdating listsRumors – when some message

received, new rumors are merged to the local list of known rumors

r0

r2

r4

r0

r2

r4

Randomized unbalanced Randomized unbalanced gossip –gossip –updating lists (cont)updating lists (cont)Active – q can be marked as faulty if:

1. Received a message where q is marked as faulty

2. q is my neighbor in communication graph and I didn’t receive a graph message from it

3. I sent a query to q and didn’t receive a reply in two rounds.

Randomized unbalanced Randomized unbalanced gossip –gossip –updating lists (cont)updating lists (cont) Pending – q can be marked as fully informed if:1. Received a message where q is

marked as fully informed2. Received a notification message

from q3. I’m a disseminator and I sent a

notification to q

notification

notification

Randomized unbalanced Randomized unbalanced gossip -gossip -correctnesscorrectnessClaim: there is at least one

leader that never fails if t<n◦If 2t<n than m=2t hence at least

half of the leaders won’t fail◦If 2t>n than m=n hence at least one

leader won’t failConclusion: at least one leader

will run an ending phase. During the phase it will learn about all processors and will disseminate this knowledge

Randomized unbalanced Randomized unbalanced gossip -gossip -complexitycomplexityRp – is a conceptual list of

chunks that has at least one node that p has not hear about

rk(i) – let K be subgraph of

Sp – is a conceptual list of chunks that has at least one node that p has to notify

sk(i) – let K be subgraph of

( )K pp K

r i R

( )lG m

( )lG m

( )K pp K

s i S

Randomized unbalanced Randomized unbalanced gossip -gossip -complexitycomplexityLook on the graph formed

by m leaders. At least m-t leaders never fail

there are at least (m-t)/7 nodes in connected component with radius 2+30ln(m). Let call this subgraph K

( ) propartieslG m

( )lG m

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)Lemma: if a stage takes

phases then

With probability that at least

Proof: If chunk is not in all Rp lists it will be removed from all other lists in one stage.

The worst case is when all chunks are in all lists

c 2 30ln m

( 1) max ( 1) ln , ( 1) / 2r i r i K m r i

1 exp ln / 30c K m

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)Let us consider the choices of

chunks made by the processors in K as occurring sequentially.

Consider a sequence of 30*|K|*ln(m) consecutive trials X1,X2, . . . , which represents the case of c = 1

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)Case |K|ln(m)>r(i-1)/2Let us consider it to be a success

in trial Xi if either a new chunk is selected or the number of chunks selected already by this trial is at least r(i − 1)/2.

The probability of success in a trial is at least 1/2

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)Case |K|ln(m) ≤ r(i-1)/2Let us consider it to be a success

in trial Xi if either a new chunk is selected or the number of chunks selected already by this trial is at least |K|ln(m) .

The probability of success in a trial is at least 1/2

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)In both cases we have 30*|K|

*ln(m) Bernouli trials with probability at least ½ for success

2

Chernof bound:

Pr[B (1 ) exp( / 2)

115 | | ln ; 15

k

K m

( 1) max ( 1) ln , ( 1) / 2r i r i K m r i

With probability that at least

1 exp ln / 30c K m

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)Lemma For each , there is

such that gossiping is successfully completed by regular phase , while communication is

by this phase, with probability that is at least

0 0

lg(w)*ln(m)

13.52log 742log

nO n t

n t

1 exp( m t lnm)

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)Proof:

Let L be a fixed subgraph of induced by m−t vertices.

There are O(1) stages that

There is at most 1+log(w) stages that

Other states are called useless

( )lG m

( 1) ( 1) ln7

m tr i r i m

( 1)( 1)

2

r ir i

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)there is a constant β > 0 that if

there is no useless stage among the first even

β +lg(w) ones, then r(β +lg(w)) = 0.The probability that there is a useless stages is:There are at most

subgraphs L

( lg ) exp ( ) ln / 30w c m t m

| |

m tm m em

L m t m t

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)

Hence the probability that there is a useless stage among the first even

β + lg(w) ones, for an arbitrary subgraph L

lg( ) exp ( ) ln( ) exp - (m-t)ln(m) / 230

em cw m t m

m t

This is a probability that some collector

didn’t become a disseminator

Randomized unbalanced Randomized unbalanced gossip -gossip -complexity (cont)complexity (cont)We have the same probability that if

after β + lg(w) even stages all leaders became a collectors, but in following β + lg(w) even stages there is some uninformed node.

The probability that there is a disseminator after 4 (β + lg(w))

stages is exp - (m-t)ln(m)

Randomized unbalanced Randomized unbalanced gossip -gossip -time complexitytime complexity stagesEach stage takes

phases

phases

4 + lg(w)

c · 2 + 30 ln(m)

( )m O t

( )w O t

2(log )O t

Randomized unbalanced Randomized unbalanced gossip -gossip -message complexitymessage complexity

Number of graph messages

Number of inquiry messages

Message complexity

2(m leaders) (O(log t) phases) (maximum degree of the communication graph)

2 2(m leaders) (O(log t) phases) (the size of a chunk)=O n·log (t)· n-t

n

27/22log 74m

Om t

17/52log

nO n t

n t

From random to From random to deterministic -deterministic - unbalanced gossip unbalanced gossip Take number such that . Let α > 0 be the corresponding number

that there exists a family of local permutations Π such that termination threshold T = α lgwln(m) = guarantees completion of gossiping without invoking ending phases. Make this threshold

value T and such family Π a part of code of algorithm UNBALANCED-GOSSIP

UNBALANCED-GOSSIP has time complexity , and message complexity

If then we get : time complexity , and

message complexity

2O(log t)17/5

2O n log tn

n t

exp m t ln(m) 1

2(log )O t

( / log )an t n n 2O(log t)2 17 /5O( log t log )an n

∏ is only

proved to exist

If a=0 then message complexity

2O( log t)n

ConsensusConsensusEvery process starts with some initial

value {0,1}Processor decides on its decision value

Termination: Each processor eventually chooses a decision value, unless it crushes

Agreement: No two processors choose different decision values

Validity: Only a value among the initial ones may be chosen as a decision values

ConsensusConsensusgossip consensusLet gossip and than decide on the

maximum value……What if some processor has crushed

and its input value may be known only to subset of processors?

Gossip can be solved in O(1) time while consensus with failures can’t be solved in less than t+1 rounds.

?

ConsensusConsensusThe algorithm is designed for t

failuresIf Time complexity: Communication complexity:

( )O t2( log )O n t

( )n t n

Consensus – Consensus – White knights consensusWhite knights consensus

Leaders reach a consensus and then tell their decision to regular nodes.

Leaders send messages along a communication graph

In order to handle partition of ,in case of failure, nodes run gossip algorithm

( )lG m

( )lG m

Consensus – Consensus – White knights consensusWhite knights consensus1) Set to initial value2) Repeat times

a) if then send message to every neighborb) Repeat m times

a) Receive short messagesb) If and received message than setc) If was set to1 in this round, send message to all

neighborsc) Repeat 2+30log(m) times

a) Receive compactness messagesb) Merge Nearby lists c) Send compactness messages to all neighbors

d) If Nearby list containing less than (m-t)/7 nodes, set

e) Perform gossiping

3) Decide on value

prumor7 / ( ) 1m m t

1prumor

0prumor 1prumor

prumor

0prumor

prumor

Consensus – Consensus – White knights consensusWhite knights consensus

Send your preference

Check compactness

Gossip

White knights consensus – White knights consensus – intuitionintuitionIn every round

If nodes preference value is 1 it sends it to its neighbors. If it received new preference value 1 it sends it to its neighbors

This is done for m rounds to ensure that ‘1‘ propagates to all nodes in the connected component

All nodes in connected component has the same rumor value after step 2.b.b

White knights consensus – White knights consensus – intuitionintuitionwhy so many phaseswhy so many phases??Let look on some node that before gossiping has rumor = 1

According to the algorithm it’s connected component contains at least(m-t)/7 nods and they should have rumor = 1too.

What if they all crash while gossiping?after gossiping some nodes may have rumor = 1 but others don’t

White knights consensus – White knights consensus – intuitionintuition why so many phases? (cont)why so many phases? (cont)If this scenario happens every

iteration than nodes won’t reach consensus.

every iteration at least (m-t)/7 nodes should fail

It is impossible that all m nodes crush 7

( ) / 7m

Itr m t m Itrm t

White knights consensus - White knights consensus - correctnesscorrectnessA processor is said to be a white

knight in an iteration, if it starts gossiping in this iteration with the rumor equal to 1.

White knights consensus - White knights consensus - correctness(cont)correctness(cont)

The decision value is among input values

all inputs are“0” - no “1“ ever appearall inputs are“1” – At least one

processor that never fails will stay compact through all iterations and it will spread its “1” value at the last gossiping step

White knights consensus - White knights consensus - correctness(cont)correctness(cont)

All processors decide on the same value

There is an iteration without white knight all nodes have rumor “0”

There is an iteration with white knight and it survives gossiping – every processor learns its rumor and next iteration all processors will start with rumor=1. At least one processor will stay compact through all iterations and it will spread its “1” value at the last gossiping step.

White knights consensus - White knights consensus - correctness(cont)correctness(cont)There are white knights in each

iteration, but no white knight survives gossiping in any iteration

We have shown that it can’t happen since there are “to many” iterations

White knights consensus – White knights consensus – time complexitytime complexity

Phases number

Phases number

m min 2t,n

/ ( )O m m t

( )n t n

(1)O

White knights consensus – White knights consensus – time complexitytime complexity1) Set to initial value2) Repeat times

a) if then send message to every neighborb) Repeat m times

a) Receive short messagesb) If and received message than setc) If was set to1 in this round, send message to all

neighborsc) Repeat 2+30log(m) times

a) Receive compactness messagesb) Merge Nearby lists c) Send compactness messages to all neighbors

d) If Nearby list containing less than (m-t)/7 nodes, set

e) Perform gossiping

3) Decide on value

prumor7 / ( ) 1m m t

1prumor

0prumor

prumor

0prumor

prumor

O(1) O(m)=O(t)

2(log )O t

(log )O t

2log logO t t t O t

White knights consensus – White knights consensus – communication complexitycommunication complexity1) Set to initial value2) Repeat times

a) if then send message to every neighborb) Repeat m times

a) Receive short messagesb) If and received message than setc) If was set to1 in this round, send message to all

neighborsc) Repeat 2+30log(m) times

a) Receive compactness messagesb) Merge Nearby lists c) Send compactness messages to all neighbors

d) If Nearby list containing less than (m-t)/7 nodes, set

e) Perform gossiping

3) Decide on value

prumor7 / ( ) 1m m t

1prumor

0prumor

prumor

0prumor

prumor

O(1) O(m)=O(t)

Every processor sends message at

least onceGraph ' s degree is

constant

Every processor sends message at

least onceGraph ' s degree is

constant

2( log )O m t

( log )O m t

2 2log log logO m m t m t O n t

Distributed computationDistributed computationThe DO-ALL problemThe DO-ALL problem

There are n processorsAt most t processors may crush.

t<nThere are j jobs to perform

◦jobs are idempotent, i.e., executing task many times and/or concurrently has the same effect as executing the task ones

Distributed computationDistributed computationThe DO-ALL problemThe DO-ALL problem

The goal is to perform all the jobs, every job should be executed at least once by some processor.

The algorithm terminates when all non faulty processors are aware that all tasks are done

Trivial solution: every processor executes all the jobs.

The DO-ALL problemThe DO-ALL problem Performance metrics Performance metricsMessage complexity – number of point to

point messages sent during the executionWork complexity – we assume that a

processor performs a unit of work per unit of time. Note that the idling processors consume a unit of work per step. For a n-processor, j-task computation subject to a failure pattern F denote by Pi(j, n, F) the number of processors that survive step i of the computation.

:

( , , ) ( , , )max iF F t

W j n F p j n F

The DO-ALL problemThe DO-ALL problemtrivial approachtrivial approach

every processor executes all the jobs

No message complexityWork complexity

To achieve better work complexity we trade messages for communication steps

( )j n

The DO-ALL problemThe DO-ALL problemThe algorithmThe algorithm

Complexity:If

we have:

Can be implemented with other gossiping algorithm achieving

3

1 2

log

for any >0

W O j n n

M O n

3

2 2

log

log ( ) log ( )

W O j n n

M O n n t

( )n t n

The DO-ALL problemThe DO-ALL problemThe algorithmThe algorithm

Taskv – list of j tasks ordered according to some permutation πv

Procv – list of processors v believes are non faulty

Donev – variable indicating whether all jobs are done according to v

The DO-ALL problemThe DO-ALL problemThe algorithmThe algorithm1. done=false2. task={π(1),π(2),…, π(j)}3. proc={1,2,…,n}4. Repeat

Repeat ßlog(n)+1 times

1. Repeat times

If task not empty Perform task whose id is first in task and remove it from

task else

set done to true

1. Run gossip with rumor=(task,proc,done)2. If done=true AND done is true for all received rumors

TERMINATEelse

update task ,proc

3log

log2

l

l

j n nT

nn

The DO-ALL problemThe DO-ALL problemThe algorithmThe algorithm

1,2,3,4,5,6,7,8

Work

1,2,3,4,5,6,7,8

1,2,3,4,5,6,7,8

Gossip

1,4,5,8

1,4,5,8

1,4,5,8

work gossip

Done

Done

Done

work

The DO-ALL problemThe DO-ALL problemThe algorithm correctnessThe algorithm correctness

If some job removed from some of the task lists it means that it was executed

The progression in each node is insured by the algorithm

Every node will finally terminate and all jobs will be executed

The DO-ALL problemThe DO-ALL problemThe algorithmThe algorithm

Complexity:If

we have:

Can be implemented with other gossiping algorithm achieving

3

1 2

log

for any >0

W O j n n

M O n

3

2 2

log

log ( ) log ( )

W O j n n

M O n n t

( )n t n

summarysummary Gossip algorithms

time complexity=Communication complexity=

◦ Randomized unbalanced gossipwill have the above complexity with high probability

◦ unbalanced gossip will have the above complexity, but it is not constructive

Consensustime complexity=message complexity=

Distributed computing – the DO-ALL problemWork complexity= Message complexity=

2(log )O t17/5

2logn

O n tn t

( )O t2( log )O n t

3logO j n n 1 2 for any >0O n

ReferencesReferencesBogdan S. Chlebus, Dariusz R.

Kowalski: Robust gossiping with an application to consensus.

J. Comput. Syst. Sci. (JCSS) 72(8):1262-1281 (2006)

Chryssis Georgiou, Dariusz R. Kowalski, and Alexander A. Shvartsman:Efficient gossip and robust distributed computationTheoretical Computer Science, Vol. 347(1-2), November 2005, pp. 130-166