![Page 1: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/1.jpg)
of 35 05/16/2012 CTW: Communication and Computation 1
Communication amid Uncertainty
Madhu Sudan Microsoft, Cambridge, USA
Based on: Universal Semantic Communication – Juba & S. (STOC 2008) Goal-Oriented Communication – Goldreich, Juba & S. (JACM 2012) Compression without a common prior … – Kalai, Khanna, Juba & S. (ICS 2011)
![Page 2: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/2.jpg)
of 35
Overview
I. Motivations/Ramblings
II. Example: Compression
III. General “Goal-oriented communication”
05/16/2012 CTW: Communication and Computation 2
![Page 3: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/3.jpg)
of 35 05/16/2012 CTW: Communication and Computation 3
The Meaning of Bits
Is this perfect communication?
What if Alice is trying to send instructions? In other words … an algorithm Does Bob understand the correct algorithm? What if Alice and Bob speak in different
(programming) languages?
Channel Alice Bob 01001011 01001011
Bob Freeze!
![Page 4: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/4.jpg)
of 35
Shannon on Semantics
05/16/2012 CTW: Communication and Computation 4
Shannon ‘48
“The semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages”
Claim: Today, the semantics are becoming important to engineering.
![Page 5: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/5.jpg)
of 35
Importance of semantics
Why is semantics (relatively) important today? Factor 1: Success of the Shannon program:
Reliability, in syntactic sense, has been achieved.
Factor 2: Communication vs. Computing.
05/16/2012 CTW: Communication and Computation 5
![Page 6: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/6.jpg)
of 35
Communication vs. Computation Interdependent technologies: Neither can exist without other
Technologies/Products/Commerce developed (mostly) independently. Early products based on clean abstractions of the other. Later versions added other capability as afterthought. Today products … deeply integrated.
Deep theories:
05/16/2012 CTW: Communication and Computation 6
Time for the theoretical wall to come down?
Well separated … and have stayed that way
Turing ‘36 Shannon ‘48
![Page 7: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/7.jpg)
of 35
Consequences of the wall
Computing theory: Fundamental principle = Universality You can program your computer to do whatever you want.
Communication principle: Centralized design (Encoder, Decoder, Compression, IPv4, TCP/IP). You can NOT program your device!
Contradiction! But does it matter? Yes! As in semantics
05/16/2012 CTW: Communication and Computation 7
![Page 8: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/8.jpg)
of 35
Role of theory?
Ideally: Foundations of practice!
05/16/2012 CTW: Communication and Computation 8
Theory layer
Application
![Page 9: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/9.jpg)
of 35
Option 1
Communication vs. Computing
05/16/2012 CTW: Communication and Computation 9
Communication
Computing
![Page 10: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/10.jpg)
of 35
Option 2
Communication vs. Computing
05/16/2012 CTW: Communication and Computation 10
Communication
Computing
![Page 11: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/11.jpg)
of 35
Option 3
Communication vs. Computing
05/16/2012 CTW: Communication and Computation 11
Communication
Computing
![Page 12: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/12.jpg)
of 35
Good News/ Bad News
Good: We are mostly practicing option 2 or 3!
Bad: Lost opportunities. Vulnerabilities. Inefficiency. Incompatibilities.
05/16/2012 CTW: Communication and Computation 12
![Page 13: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/13.jpg)
of 35
Sample problems:
Universal printing: You are visiting a new university. Can your
machine not learn how to print on the local printer, without requiring installation?
Projecting from your laptop: Machines that learn to communicate, and learn
to understand each other. Digital libraries:
Data that lives forever (communication across time), while devices change.
05/16/2012 CTW: Communication and Computation 13
![Page 14: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/14.jpg)
of 35
Essence of “semantics”: Uncertainty
Recall Shannon: “The significant aspect is that the actual
message is one selected from a set of possible messages”
Essence of unreliability today: Sender and receiver in disagreement about set
of possible messages (or about their meaning).
05/16/2012 CTW: Communication and Computation 14
![Page 15: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/15.jpg)
of 35
Modelling uncertainty
Classical Shannon Model
05/16/2012 CTW: Communication and Computation 15
A B Channel
B2
Ak
A3
A2
A1 B1
B3
Bj
Semantic Communication Model
New Class of Problems New challenges
Needs more attention!
[Kalai,Khanna,J.,S. – ICS 2011] Compression in this setting: Leads to ambiguous, redundant compression
![Page 16: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/16.jpg)
of 35 05/16/2012 CTW: Communication and Computation 16
II: Non-interactive communication: Compression
![Page 17: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/17.jpg)
of 35
Motivation: Human Communication
Human communication (dictated by languages, grammars) very different. Grammar: Rules, often violated. Dictionary: Often multiple meanings to a word. Redundant: But not as in any predefined way
(not an error-correcting code). Our thesis: Emerges from uncertainty:
Sender of message uncertain about receiver’s background/context/prior.
Will try to explain in the context of Redundancy
05/16/2012 CTW: Communication and Computation 17
![Page 18: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/18.jpg)
of 35
Behavioral aspects of natural communication
(Vast) Implicit context. Sender sends increasingly long messages to
receiver till receiver “gets” (the meaning of) the message.
Sender may use feedback from receiver if available; or estimates receiver’s knowledge if not.
Language provides sequence of (increasingly) long ways to represent a message.
Question: What is the benefit of choosing short/long messages?
05/16/2012 CTW: Communication and Computation 18
![Page 19: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/19.jpg)
of 35
Model:
Reason to choose short messages: Compression. Channel is still a scarce resource; still want to
use optimally.
Reason to choose long messages (when short ones are available): Reducing ambiguity. Sender unsure of receiver’s prior (context).
(“uncertainty”) Sender wishes to ensure receiver gets the
message, no matter what its prior (within reason).
05/16/2012 CTW: Communication and Computation 19
![Page 20: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/20.jpg)
of 35
Model
Wish to design encoding/decoding schemes (E/D) to be used as follows: Sender has distribution P on M = {1,2,…,N} Receiver has distribution Q on M = {1,2,…,N} Sender gets 𝑋 ∈ 𝑀 Sends E(P,X) to receiver. Receiver receives Y = E(P,X) Decodes to 𝑋� = D(Q,Y)
Want: X = 𝑋� (provided P,Q close),
While minimizing 𝐸𝐸𝑝𝑋←𝑃 |𝐸(𝑃,𝑋)|
05/16/2012 CTW: Communication and Computation 20
![Page 21: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/21.jpg)
of 35
Contrast with some previous models
Universal compression? Doesn’t apply: P,Q are not finitely specified. Don’t have a sequence of samples from P; just
one! K-L divergence?
Measures inefficiency of compressing for Q if real distribution is P.
But assumes encoding/decoding according to same distribution Q.
Semantic Communication: Uncertainty of sender/receiver; but no special
goal. 05/16/2012 CTW: Communication and Computation 21
![Page 22: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/22.jpg)
of 35
Closeness of distributions:
P is 𝛼-close to Q if for all 𝑋 ∈ 𝑀,
1𝛼≤ 𝑃 𝑋
𝑄 𝑋≤ 𝛼
P 𝛼-close to Q ⇒ 𝐷(𝑃||𝑄),𝐷(𝑄| 𝑃 ≤ log𝛼 .
05/16/2012 CTW: Communication and Computation 22
![Page 23: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/23.jpg)
of 35
Dictionary = Shared Randomness?
Modelling the dictionary: What should it be?
Simplifying assumption – it is shared randomness, so …
Assume sender and receiver have some shared randomness R and X is independent of R. Y = E(P,X,R) 𝑋� = D(Q,Y,R)
Want ∀𝑋, Pr
𝑅𝑋� = 𝑋 ≥ 1 − 𝜖
05/16/2012 CTW: Communication and Computation 23
![Page 24: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/24.jpg)
of 35
Solution (variant of Arith. Coding)
Use R to define sequences 𝑅1 1 ,𝑅1 2 ,𝑅1 3 , … 𝑅2 1 ,𝑅2 2 ,𝑅2 3 , … … 𝑅𝑁 1 ,𝑅𝑁 2 ,𝑅𝑁 3 , … .
𝐸𝛼 𝑃, 𝐸,𝑅 = 𝑅𝑥 1 … 𝐿 , where 𝐿 chosen s.t. ∀𝑧 ≠ 𝐸 Either 𝑅𝑧 1 … 𝐿 ≠ 𝑅𝑥 1 … 𝐿
Or 𝑃 𝑧 < 𝑃 𝑥𝛼2
𝐷𝛼 𝑄,𝑦,𝑅 = argmax𝑥� {𝑄 𝐸� } among 𝐸� ∈ 𝑧 𝑅𝑧[1 … 𝐿] = 𝑦
05/16/2012 CTW: Communication and Computation 24
![Page 25: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/25.jpg)
of 35
Performance
Obviously decoding always correct.
Easy exercise:
Exp𝑋 𝐸 𝑃,𝑋 = 𝐻 𝑃 + 2 log 𝛼
Limits: No scheme can achieve 1 − 𝜖 ⋅ [𝐻 𝑃 + log 𝛼] Can reduce randomness needed.
05/16/2012 CTW: Communication and Computation 25
![Page 26: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/26.jpg)
of 35
Implications
Reflects the tension between ambiguity resolution and compression. Larger the 𝛼 ((estimated) gap in context),
larger the encoding length. Coding scheme reflects the nature of human
process (extend messages till they feel unambiguous).
The “shared randomness’’ is a convenient starting point for discussion Dictionaries do have more structure. But have plenty of entropy too. Still … should try to do without it.
05/16/2012 CTW: Communication and Computation 26
![Page 27: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/27.jpg)
of 35 05/16/2012 CTW: Communication and Computation 27
III: Interactive Setting: Goal-Oriented Communication
![Page 28: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/28.jpg)
of 35
Back to meaning
What if sender is sending instructions? Can we ensure receiver decodes the right
instructions? Freeze?
Translation of bits to instructions? Well studied in language/computer science. (Many) “Complete” languages/codebooks exist.
Each translates bits to meaning. All equivalent (upto “Kolmogorov constant”) But not same.
05/16/2012 CTW: Communication and Computation 28
![Page 29: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/29.jpg)
of 35
Goal of communication
Easy negative result: (Due to plethora of languages/codebooks): In finite time, can’t
guarantee “receiver understands instructions.” Is this bad?
If receiver can not distinguish correct instructions from incorrect ones, why should it try to do so?
Goals of communication: Communication is not an end in itself, it a means to
achieving some end. Semantic communication:
Help communication achieve its goal. Use progress towards goal to understand meaning.
05/16/2012 CTW: Communication and Computation 29
![Page 30: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/30.jpg)
of 35
Computation as a goal [Juba+S., 2008]
The lens of computational complexity: To prove some resource is useful:
Step 1: Identify hardest problems one can solve without the resource.
Step 2: Show presence of resource can help solve even harder problems.
Classical resources: CPU speed, Memory, Non-determinism, Randomness …
In our case: Communication in presence of understanding. Communication w/o understanding.
05/16/2012 CTW: Communication and Computation 30
![Page 31: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/31.jpg)
of 35
Computation as a goal
Model: Simple user talking to powerful server. Class of problems user can solve on its own:
~ probabilistic polynomial time (P). Class of problems user can solve with perfect
understanding of server: ~ Any problem. (Even uncomputable!)
Class of problems user can solve without understanding of server: ~ Polynomial space.
Roughly: If you are solving problems and can verify solutions, then this helps. If you have a solution, you are done. If not, you’ve found some error in communication.
Moral: Communication helps, even with misunderstanding, but misunderstanding introduces limits.
05/16/2012 CTW: Communication and Computation 31
![Page 32: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/32.jpg)
of 35
Summarizing results of [GJS 2012]
But not all goals are computational. We use communication mostly for (remote) control. Intellectual/informational goals are rare(r).
Modelling general goals, in the presence of misunderstanding: Non-trivial, but can be done. Results extend those from computational setting:
Goals can be achieved if user can sense progress towards goal, servers are “forgiving” and “helpful”
05/16/2012 CTW: Communication and Computation 32
![Page 33: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/33.jpg)
of 35
Useful lessons
User/Server can be designed separately.
Each should attempt to model its “uncertainty” about the other.
Each should plan for uncertainty: Server: By assuming some short “interrupt”
sequence. User: By always checking its progress.
05/16/2012 CTW: Communication and Computation 33
![Page 34: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/34.jpg)
of 35
Future goals
Broadly: Information-theoretic study of human
communication, with uncertainty as an ingredient. Should exploit natural restrictions of
humans: Limited ability to learn/infer/decode. Limited bandwidth.
Conversely, use human interactions to create alternate paradigms for “designed communications. Place semantics on solid foundations.
05/16/2012 CTW: Communication and Computation 34
![Page 35: Communication amid Uncertaintypeople.seas.harvard.edu/~madhusudan/talks/2012/UncertainComm.pdf · 05/16/2012 CTW: Communication and Computation 1 of 35 Communication amid Uncertainty](https://reader033.vdocuments.us/reader033/viewer/2022041521/5e2e5708745c8a6d9a2a21ba/html5/thumbnails/35.jpg)
of 35 05/16/2012 CTW: Communication and Computation 35
Thank You!