Towards a Theory of Semantic Communication

Jie Bao, RPI

Joint work with Prithwish Basu, Mike Dean, Craig Partridge, Ananthram

Swami, Will Leland and Jim Hendler 1

Outline

• Background• A general semantic communication model• Measuring semantics• Semantic data compression (source coding)• Semantic reliable communication (channel

coding) • Path ahead

2

Shannon, 1948

“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;... These semantic aspects of communication are irrelevant to the engineering problem.”

3Claude E. Shannon. A mathematical theory of communication. Bell System Technical Journal, 27:379-423, 625-56, 1948.

message

message

Signal

Signal

But, are these just sequences of bits?

• Movie streams• Software codes• DNA sequences• Emails• Tweets• ……

4

“The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning;..”“These semantic aspects of communication are irrelevant to the engineering problem”?

Between a Talent Manager & Me

“Are you open to discuss greener pastures”?

5

“Thanks for contacting me. However, I'm not sure if my research is related to "greener pastures". I'm a computer scientist.”

Misunderstanding can be costly

6

Mars Climate Orbiter (1998-1999), $125 million

Expressed

Pound (lbF)

Interpreted

Newton (N)

Image Source: Wikipedia, http://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Communications_loss

Misunderstanding can be deadly

Afghan National Army (ANA) to ISAF• “Launch flares over the left side of the village”

Received and Understood as• “fire on the left side of the village”

Alternative semantic coding (e.g., illuminating shell) may save lives!

7Scenario based on report from http://www.closeprotectionworld.co.uk/security-news-asia/37466-afghanistan-war-what-happens-when-war-interpreter-doesnt-know-language.html

(Noisy) Battlefield Communication (Noisy) Battlefield Communication

Our Contributions

• We develop a generic model of semantic communication, extending the classic model-theoretical work of (Carnap and Bar-Hillel 1952) ;

• We discuss the role of semantics in reducing source redundancy, and potential approaches for lossless and lossy semantic data compression;

• We define the notions of semantic noise, semantic channel, and obtain the semantic capacity of a channel.

Outline

• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel

coding) • Path ahead

9

(Classical) Information Theory Semantic Information Theory

Shannon, 1948

message

message

Shannon ModelShannon Model

Signal

Signal

ExpressedMessage(e.g., commands and reports)

Expressed Message

Semantic Channel

From IT to SIT

A Three-level Model (Weaver)

Transmitter Receiver

Destination Destination Source Source

Physical Channel

Technical message

Technical Noise

Intended message

Expressed message

Semantic Transmitter

Semantic Transmitter

Semantic ReceiverSemantic Receiver

Semantic Noise

Semantic Noise

Shared knowledge

Shared knowledge

Local knowledge

Local knowledge

Local knowledge

Local knowledge

(effectiveness factors)

C: Effectiveness

B: Semantic

A: Technical

A Semantic Communication Model

12

Message generator

World model

Background Knowledge

Inference Procedure

Messages

Sender

Message interpreter

World model

Background Knowledge

Inference Procedure

Receiver

Ws Wr

Ks KrIs Ir

{m}

World

M: Message Syntax

Feedback (?)

observations

Ms Mr

Semantic Sources

• Key: A semantic source tells something that is “true”– Engineering bits are neither true or false!

• Goal: 1) more soundness (sent as “true”->received as “true”); 2) less ambiguity

13

Outline

• Background• A general semantic communication model• Measuring semantics• Semantic data compression (source coding)• Semantic reliable communication (channel

coding) • Path ahead

14

Measuring Semantic Information

• Basic Problem: What is the amount of “semantics” carried by a source and its messages?

15

Measuring Semantic Information

• Statistical approach: Inference may change the distribution of symbols, hence the entropy of the source.

• Model-theoretical approach: The less “likely” a message is to be true, the more information it contains.

• Algorithmic approach: What’s the minimal program needed to describe messages and their deductions?

• Situation-theoretical approach: measuring the divergence of messages to “truth”.

16

Our ApproachOur Approach

Shannon: Information = “surpriseness”

17

H(tyrannosaurus) > H(dog)H(tyrannosaurus) > H(dog)

Captured from: http://www.wordcount.org/main.php

Which sentence is more “surprising”?

18

``Rex is not a tyrannosaurus''

``Rex is not a dog''

????

Model Semantics

• tyrannosaurus • dog

19

??

“Semantics” of DNA

20Image courtesy: http://www.yourdictionary.com/dna http://www.pnl.gov/biology/images/protein_molecule.jpg

“Syntax” Model (“Semantics”)

Gene expression

Stone-age Semantic Communication

• Semantic communication predates symbolic communications

21Altamira Cave Painting http://mandyking.files.wordpress.com/2011/02/altamira-cave.jpg

Semantics of Messages

• Messages are expressions, not just sequences of symbols– E.g., Saturday->Weekend, Sunny & Cold

• If an expression is more commonly true, it contains less semantic information– inf (Sunny & Cold) > inf (Cold)– inf (Cold) > inf (Cold or Warm)

22

Semantics of Messages

• Carnap & Bar-Hillel (1952) - “An outline of a theory of semantic information”

m(exp) = |mod(exp)| / |all models|

inf(exp) = - log m(exp)

• Example– m(A v B) = ¾, m(A ^ B)=1/4– Inf(A v B)=0.415, inf(A^B )= 2

23

Knowledge Entropy

• Extending Carnap & Bar-Hillel (1952) – Models have a distribution– Background knowledge may present

Weekend=2/7, Saturday=1/7

Knowledge Entropy

• Logical prob. and knowledge entropy of Messages

• Model entropy of an information source

25

model distribution

logical probability

Semantic Information Calculator (Demo)

• http://www.cs.rpi.edu/~baojie/sit/index.php

Outline

• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel

coding) • Path ahead

27

Conditional Knowledge Entropy

• When there is background knowledge, the set of possible worlds decreases.

28

Model Compression with Shared Knlg

• Background knowledge (A->B), when shared, help compress the source– Side information in the form of entailment

29

Lossless Message Compression

• Theorem : There is a semantically lossless code for source X, with message entropy H >= H(Xeq); no such code exists for H < H(Xeq)

– Xeq are equivalent classes of X

• Example: no need for coding both “pig” and “swine”, using one of them is sufficient.

• Example 2: a->(a^b)v(b^c) = a->b• Sometime, the loss is intentional compression

– Textual description of an image– Abstract of a paper

Other Source Coding Strategies

• Lossless model compression– E.g., using minimal models

• Lossy message compression– E.g., compressing based on semantic similarity

• Leave as future work

31

Outline

• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel

coding) • Path ahead

32

Semantic Noise

Examples

• The meaning of a message is changed due to technical noises, e.g., from ``flare'' to ``fire'‘;

• Semantic mismatch: The source / receiver use different background knowledge or inference (e.g., during the loss of the Mars Climate Orbiter);

• Lost in translation: “Uncle” in English has no exact correspondence in Chinese.

33

Semantic Noise and Channel Coding

34

“coffee machine”“copy machine”

“Xerox” “Xerox”

“copy machine”

p->ff

?

?

0.9

0.1

1.0

W X Y W’

Scenario developed based on reports in http://english.visitkorea.or.kr/enu/AK/AK_EN_1_6_8_5.jsp and  http://blog.cleveland.com/metro/2011/03/identifying_photocopy_machine.html

Semantic Channel Coding Theorem

• In the simplified model, assume no semantic mismatch (Ks=Kr, Is=Ir)

• Theorem 3: If transmission rate is smaller than Cs (semantic channel capacity), error-free coding exists

• Semantic channel capacity may be higher or lower than the engineering channel capacity (sup I(X;Y)) !– H(W|X) stands for encoder’s semantic ambiguity – avg(inf(Y)) is receiver’s “smartness”

35

Outline

• Background• A general semantic communication model• Measuring Semantics• Semantic data compression (source coding)• Semantic reliable communication (channel

coding) • Path ahead

36

Application in Coding & Validation

• Hypothesis 1: using semantics we can achieve better data compression

• Hypothesis 2: using semantics we can achieve more reliable communication

• Validation with comparison to non-semantic algorithms

Extensions

• Extensions & connections to other fields

– First-order languages [probabilistic logics]– Inconsistent KBs (misinformation) [paraconsistent

logics]– Lossy source coding [clustering and similarity

measurement]– Semantic mismatches [extending Juba & Sudan

2011]– … …

Path ahead – Broad Impact

– Communications (e.g., coding)– Linguistics (e.g., entropy of English)– Biology (e.g., semantics of genes)– Economics – ….– Areas wherever Shannon’s theory applies – And beyond (e.g., Semantic Web, ontology

engineering)

Questions?

40Image courtesy: http://www.addletters.com/pictures/bart-simpson-generator/900788.htm