![Page 1: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/1.jpg)
of 2411/20/2012 TIFR: Deterministic Communication Amid Uncertainty 1
(Deterministic) Communication amid Uncertainty
Madhu SudanMicrosoft, New England
Based on joint works with:(1) Adam Kalai (MSR), Sanjeev Khanna (U.Penn), Brendan Juba
(Harvard)and (2) Elad Haramaty (Technion)
![Page 2: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/2.jpg)
of 24
Classical Communication
The Shannon setting Alice gets chosen from distribution Sends some compression to Bob. Bob computes
(with knowledge of ). Hope .
Classical Uncertainty:
Today’s talk: Bob knows .
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 2
![Page 3: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/3.jpg)
of 24
Outline
Part 1: Motivation Part 2: Formalism Part 3: Randomized Solution Part 4: Issues with Randomized Solution Part 5: Deterministic Issues.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 3
![Page 4: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/4.jpg)
of 24
Motivation: Human Communication
Human communication (dictated by languages, grammars) very different from Shannon setting. Grammar: Rules, often violated. Dictionary: Often multiple meanings to a word. Redundant: But not as in any predefined way
(not an error-correcting code).
Theory? Information theory? Linguistics? (Universal grammars etc.)?
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 4
![Page 5: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/5.jpg)
of 24
Behavioral aspects of natural communication
(Vast) Implicit context. Sender sends increasingly long messages to
receiver till receiver “gets” (the meaning of) the message. Where do the options come from?
Sender may use feedback from receiver if available; or estimates receiver’s knowledge if not. How does estimation influence message.
Language provides sequence of (increasingly) long ways to represent a message. How? Why? What features are good/bad.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 5
![Page 6: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/6.jpg)
of 24
Model:
Reason to choose short messages: Compression. Channel is still a scarce resource; still want to
use optimally.
Reason to choose long messages (when short ones are available): Reducing ambiguity. Sender unsure of receiver’s prior (context). Sender wishes to ensure receiver gets the
message, no matter what its prior (within reason).
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 6
![Page 7: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/7.jpg)
of 24
Back to Problem
Design encoding/decoding schemes () so that Sender has distribution on Receiver has distribution on Sender gets Sends to receiver. Receiver receives Decodes to
Want: (provided close), While minimizing
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 7
![Page 8: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/8.jpg)
of 24
Contrast with some previous models
Universal compression? Doesn’t apply: P,Q are not finitely specified. Don’t have a sequence of samples from P; just
one! K-L divergence?
Measures inefficiency of compressing for Q if real distribution is P.
But assumes encoding/decoding according to same distribution Q.
Semantic Communication: Uncertainty of sender/receiver; but no special
goal.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 8
![Page 9: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/9.jpg)
of 24
Closeness of distributions:
is -close to if for all , -close to (symmetrized, “worst-case” KL-divergence)
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 9
![Page 10: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/10.jpg)
of 24
Dictionary = Shared Randomness?
Modelling the dictionary: What should it be?
Simplifying assumption – it is shared randomness, so …
Assume sender and receiver have some shared randomness and are independent of .
Want
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 10
![Page 11: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/11.jpg)
of 24
Solution (variant of Arith. Coding)
Use to define sequences
…
where chosen s.t. Either Or
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 11
![Page 12: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/12.jpg)
of 24
Performance
Obviously decoding always correct.
Easy exercise:
( “binary entropy”)
Limits: No scheme can achieve Can reduce randomness needed.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 12
![Page 13: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/13.jpg)
of 24
Implications
Reflects the tension between ambiguity resolution and compression. Larger the ((estimated) gap in context), larger
the encoding length. Coding scheme reflects the nature of human
process (extend messages till they feel unambiguous).
The “shared randomness’’ is a convenient starting point for discussion Dictionaries do have more structure. But have plenty of entropy too. Still … should try to do without it.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 13
![Page 14: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/14.jpg)
of 24
Deterministic Compression?
Randomness fundamental to solution. Needs independent of to work.
Can there be a deterministic solution? Technically: Hard to come up with single
scheme that compresses consistently for all (). Conceptually: Nicer to know “dictionary” and
context can be interdependent.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 14
![Page 15: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/15.jpg)
of 24
Challenging special case
Alice has permutation on i.e., 1-1 function mapping
Bob has permutation Know both are close:
(say ) Alice and Bob know (say ).
Alice wishes to communicate to Bob. Can we do this with few bits?
Say bits if .
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 15
![Page 16: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/16.jpg)
of 24
Consider family of graphs Vertices = permutations on Edges = close permutations with distinct
messages. (two potential Alices).
Central question: What is ?
Model as a graph coloring problem
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 16
𝜋
𝜎X
![Page 17: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/17.jpg)
of 24
Main Results [w. Elad Haramaty]
Claim: Compression length for toy problem Thm 1:
( times) .
Thm 2: uncertain comm. schemes with1. (-error).2. -error).
Rest of the talk: Graph coloring
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 17
![Page 18: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/18.jpg)
of 24
Restricted Uncertainty Graphs
Will look at Vertices: restrictions of permutations to first
coordinates. Edges:
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 18
𝜋
𝜎X
𝜋 ′
𝜎 ′
![Page 19: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/19.jpg)
of 24
Homomorphisms
homomorphic to () if Homorphisms?
is -colorable
Homomorphisms and Uncertainty graphs.
Suffices to upper bound
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 19
![Page 20: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/20.jpg)
of 24
Chromatic number of
For , Let
Claim: is an independent set of
Claim:
Corollary:
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 20
![Page 21: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/21.jpg)
of 24
𝐻𝐺
Better upper bounds:
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 21
Say
Lemma:
For
![Page 22: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/22.jpg)
of 24
Lemma: For Corollary:
Aside: Can show: Implies can’t expect simple derandomization of
the randomized compression scheme.
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 22
Better upper bounds:
![Page 23: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/23.jpg)
of 24
Future work?
Open Questions: Is ? Can we compress arbitrary distributions to ? or
even On conceptual side:
Better mathematical understanding of forces on language.
Information-theoretic Computational Evolutionary
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 23
![Page 24: Of 24 11/20/2012TIFR: Deterministic Communication Amid Uncertainty1 ( Deterministic ) Communication amid Uncertainty Madhu Sudan Microsoft, New England](https://reader038.vdocuments.us/reader038/viewer/2022110304/551c319f550346a34f8b63ad/html5/thumbnails/24.jpg)
of 24
Thank You
11/20/2012 TIFR: Deterministic Communication Amid Uncertainty 24