The Role of Communication Complexity in Distributed Computing
Rotem Oshman
Background: Distributed Computing
Distributed Computing
• Typical model:– Local computation for free– Charge for “communication”
Distributed Lower Bounds
• “All or nothing”:– If have not communicated, they know nothing
about each other– If communicated, they know everything about
each other• Recently: interest in quantifying the amount of
communication• Natural to use communication complexity
Shared Memory
• Processes communicate by accessing objects in shared memory– Read/write registers– Read-modify-write: CAS, T&S, …
• Typically asynchronous:– Schedule = sequence of process IDs– Adversarially chosen– Sometimes processes may crash
Shared Memory Lower Bounds
• “Covering arguments”• Examples:– Deterministic consensus impossible if one process
may fail [FLP’79]– Mutual exclusion requires shared registers [BL’90]
• Can we introduce a cell-probe-like model for shared memory lower bounds?
Message Passing
• Processes communicate by sending messages– Over some network graph, often complete
• Fully synchronous, fully asynchronous, or anywhere in between
• Processes can crash, recover, cheat, lie,…
• Many successful applications of CC
Some differences…
Complexity measure
Problems
Input
Distributed Computing Comm. Complexity#rounds (limited bandwidth) total #bits
Search Decision
Number-In-Hand Number-on- Forehead (usually)
Message-Passing Models
MESSAGE-PASSING
LOCAL
CONGEST
SHARED BLACKBOARD#rounds total CC
Talk Overview
I. Lower bound techniquesa. CONGEST (#rounds): reductions from 2-party
communication complexityb. Total CC with private channels
II. Shared blackboarda. Number-in-handb. “Not-quite-number-in-hand”
The CONGEST Model
• nodes communicate over a network graph (small diameter)
• Computation proceeds in synchronous rounds• Each message = bits• Several variants:– Communication by unicast vs. local broadcast– Arbitrary graph vs. complete graph
The CONGEST Model
• Arbitrary graph topology: strong lower bounds– [Distributed Verification and Hardness of Distributed
Approximation, by Das Sarma, Holzer, Kor, Korman, Nanongkai, Pandurangan, Peleg, Wattenhofer]• Nearly-tight lower bound on MST approximation• Non-tight lower bounds on all-pairs shortest path,
minimum cut, shortest s-t path, …• Almost all lower bounds
– Very few bounds known, no super-linear bounds
The CONGEST Model
• Complete graph topology, unicast:– Known to be extremely powerful, e.g.,• Sort values in rounds• MST construction in rounds
– No explicit super-constant lower bound known– [Drucker, Kuhn, O. ‘14]: even slightly super-
constant lower bound new ACC lower bound• Complete graph topology, broadcast = shared
blackboard
CONGEST Lower Bounds for Arbitrary Graphs
… by reduction from 2-party disjointness
2-Party Reductions
• Textbook reduction [Kushilevitz Nisan]:Given algorithm for solving task …
Solution for answer for, e.g., Disjointness
bits
𝑌𝑋
Based on
Based on
Simulate
2-Party Reductions
• More generally:Given algorithm for solving task …
Solution for answer for, e.g., Disjointness
𝑌𝑋
Based on
Based on
Multi-Player NIH Communication with Private Channels
The Message-Passing Model
• players• Private channels• Private -bit inputs • Private randomness
• Goal: compute • Cost: total communication
The Coordinator Model
• players, one coordinator• The coordinator has no input
Message-Passing vs. Coordinator
≈
Secure multi-party computation!
Message-Passing Lower Bounds
For players with -bit inputs…• Woodruff, Zhang ‘12: estimation• Phillips, Verbin, Zhang ’12:– for bitwise problems (AND/OR, MAJ, …)
• Woodruff, Zhang ‘13:– for threshold and graph problems
• Braverman, Ellen, O., Pitassi, Vaikuntanathan ‘13: for disjointness
Symmetrization [Phillips, Verbin, Zhang ’12]
Lower bound for -player problem :• Choose hard 2-player problem • Fix hard distribution for , let
Alice:
Bob:
“Smear” across players
Symmetrization [Phillips, Verbin, Zhang ’12]
Lower bound for -player problem :
– -player distribution must be symmetric– Answer to answer to
cost for = cost for
Alice:
Bob:
“Smear” across players
Symmetrization Example: Bitwise-XOR
• 2-party problem:– Input: uniform independent – Goal: Alice outputs Bob’s input,
• Reduction:– Sample uniform , Alice sets – Bob chooses uniformly s.t. – From , Alice can reconstruct
Alice:
Bob:
Set Disjointness
Disj𝑛 ,𝑘 =¿ 𝑖=1¿𝑛¿ 𝑗=1¿𝑘 𝑋 𝑖𝑗¿
?
𝑋 1
𝑋 2
𝑋 3
𝑋 4𝑋 5
Symmetrization vs. Disjointness
• Consider any symmetric distribution…
• How many 0s in coord. , given ?– More than one ⇒ Bob probably sees a zero• Ignore this coordinate
– Only one Pr[ Alice got it ] – 2-party CC
Alice:
Bob:
[BEOPV’13] Lower Bound Outline
1. Direct sum: 2. One-bit lower bound:
Disj𝑛 ,𝑘 =¿ 𝑖=1¿𝑛¿ 𝑗=1¿𝑘 𝑋 𝑖𝑗¿
Reduction from DISJ tograph connectivity [Based on WZ’13]
1
2
3
4
5
6
𝑝1
𝑝2
𝑝𝑘
(Players)
𝑋 𝑖
[𝑛 ]∖⋃𝑋 𝑖
(Elements)
input graph connected
Number-In-Hand Shared Blackboard
Why Should We Care?
• Some fundamental question still open• Natural model for distributed computing– Single-hop wireless network– More generally, abstracts away network topology– Related to MapReduce, etc. [Hegeman and
Pemmaraju’14]
Example: NIH Multi-Party Disjointness
• Trivial upper bound: – Also easy to get
• Simultaneous CC: [Braverman,Ellen,O.,Pitassi,Vaikuntanathan’13]
– In rounds?– Looks like for [current work with Ran Raz]
• Unbounded rounds: [with Mark Braverman]
“Not-Quite Number in Hand”
• In undirected networks, each edge is known to both endpoints
• Distributed graph property testing:– Players , input graph – Input to player : its neighbors in – Goal: test if satisfies property
• Example: subgraph detection [Drucker, Kuhn, O. ‘14]
Example: Lower Bound for
• Claim: -round algorithm for detection solve DISJ in bits
• Reduction outline:– Alice and Bob get inputs – Construct input graph on nodes, such that
contains – Simulate the run of -detection algorithm on
12
34
12
34
12
34
12
34
Construction of from
Construction of from 1
2
34
12
34
12
34
12
34
Top-bottom: Bottom-top:
12
34
12
34
12
34
12
34
Construction of from
(𝟐 ,𝟒 )∈𝑿∩𝒀
Simulating the Algorithm
• Alice simulates nodes, Bob simulates • To simulate one round, each player:– Locally computes message broadcast by each
node it simulates– Sends all messages to the other player– Cost: per round
• Total cost:
What About ?
• More complicated….Can contain regardless of
Solution: use extremal -free
graph Elements of DISJ
= edges of
Upper Bound on Subgraph Detection
• Turán number: max # edges in -free graph on vertices
• Upper bound: solve -subgraph detection in rounds– Example: , nearly tight– Open problem: is this tight for all subgraphs?
Detecting Triangles
• Trivial upper bound: rounds• Lower bound?– 2-party (black box) reduction cannot prove it– For each triangle, one player knows all 3 edges
Triangles to 3-Party NOF Disjointness
• [Ruzsa and Szemerédi ’76]: there is a tripartite graph where
– contains triangles– Each edge in belongs to exactly one triangle
• Reduce from 3-party NOF Disjointness on elements, each representing one triangle in
Triangles to 3-Party NOF Disjointness
• Input: sets of triangles • Let be the unique triangle edge belongs to• Construct , including:– iff – iff – iff
• Note: endpoints of each edge agree on its inclusion!
𝑍
𝑋
𝑌
𝐴
𝐶
𝐵
Triangles to 3-Party NOF Disjointness
• Input: sets of triangles • Triangle appears in • Cost of simulation:– bits per round⇒ Round complexity of triangle detection
for 3-party NOF
𝑍
𝑋
𝑌
𝐴
𝐶
𝐵
3-Party NOF Disjointness
• Randomized CC:– Sherstov’13: – We get nothing:
• Deterministic CC:– Rao and Yehudayoff’14: ⇒ for deterministic triangle detection
Conclusion
MESSAGE-PASSING
LOCAL
CONGEST
SHARED BLACKBOARD#rounds total CC
Directions for Future Research
• Exploiting asynchrony and faults to get stronger communication lower bounds
Example 1: Dynamic Networks
• Abstract model for dynamic networks:– In each round we get a different graph
• [Kuhn, Lynch, O. ‘10]:– Assume each is connected– Any function can be deterministically computed in
rounds using -bit messages– Lower bounds?
Example 1: Dynamic Networks
• For exchanging all inputs:– Determistic, “routing-based” algorithms: [Haeupler,
Kuhn’12]
– Randomized??– Non-routing based??
Example 2: Byzantine Consensus
• processes, synchronous message-passing• Each process receives a bit• Goal:– Everyone outputs the same bit– If everyone received , the output is
• Byzantine faults: up to processes may behave arbitrarily
Example 2: Byzantine Consensus
• [King, Saia ‘10]: can solve with total bits• No general lower bound better than