Chapter 7

The Channel and Mutual

Information

Information Through a Channel

alphabet of

symbols sent

a1 :

:

aq

A

b1 :

:

bs

B

alphabet of

symbols

receivedP(bj|ai)

symbols can’t be swallowed

or randomly generated

For example, in an error correcting code over a noisy channel, s ≥ q.

If two symbols sent are indistinguishable when received, s < q.

Characterize a stationary channel by a matrix of conditional probabilities:

Pi,j = P(bj | ai)

rowcolumn

received

sent

P = Pi,jq

s

out comemust some ,input each for 1)|(11

, ji

s

jij

s

jji baabPP

7.1, 7.2, 7.3

1)()|( then sent, being ofy probabilit )( if 1 1

q

i

s

jiijii apabPaap

For p(ai) = probability of source symbols, let p(bj) = probability of being received

[p(a1) … p(aq)]P = [p(b1) … p(bs)]

sjabPapbpq

iijij ,1)|()()(

1

no noise: Pi,j = I; p(bj) = p(aj)

all noise: Pi,j = 1/s; p(bj) = 1/s

The probability that ai was sent and bj was received is:

P(ai, bj) = p(ai) ∙ P(bj | ai) = p(bj) ∙ P(ai | bj). [coincidental probability]

Baye’s Theorem

So if p(bj) ≠ 0, the backwards conditional probabilities are:

.)()|(

)()|()(

)()|()|(

1

q

iiij

iij

j

iijji

apabP

apabPbp

apabPbaP

1)|(:over sum Now,1

q

iji baPi

7.1, 7.2, 7.3

Binary symmetric Channel

p(a = 0) = p

p(a = 1) = 1 − p

a = 0

a = 1

b = 0

b = 1

P0,0 = P1,1 = P

P0,1 = P1,0 = Q

P0,0

P1,1

P1,0

P0,1

P Q

Q P

(p 1−p)

a = 0 a = 1

( pP + (1 − p)Q pQ + (1 − p)P )

=

p(b = 0) p(b = 1)

7.4

Backwards conditional probabilities No noise All noise

P(a = 0 | b = 0) =Pp

1 pPp + Q(1−p)

P(a = 1 | b = 0) =Q(1−p)

0 1 − pPp + Q(1−p)

P(a = 0 | b = 1) =Qp

0 pQp + P(1−p)

P(a = 1 | b = 1) =P(1−p)

1 1 − pQp + P(1−p)

P = 1 Q = 0 P = Q = ½

If p = 1 − p = ½ (equiprobable) then:

P(a = 1 | b = 0) = P(a = 0 | b = 1) = Q

P(a = 0 | b = 0) = P(a = 1 | b = 1) = P

P

P

QQ

7.4

System Entropies

q

i ii ap

apAH1 )(

1log)()(

H(A)

Input entropy

s

j jj bp

bpBH1 )(

1log)()(

H(B)

Output entropy

condition on bj

q

i jijij baP

baPbAH1 )|(

1log)|()|(

average over all bj

s

j

q

i jiji

s

jjj baP

baPbAHbpBAH1 11 )|(

1log),()|()()|(

Similarly

q

i

s

j ijji abP

baPABH1 1 )|(

1log),()|(

The information loss in the channel, called equivocation (or noise entropy). The average uncertainty about the symbol

sent or received.

7.5

H(A| B) H(B| A)

Intuition: taking snapshots

A B

q

i

s

j jiji baP

baPBAH1 1 ),(

1log),(),(

q

i

s

j ijji

q

i

s

j iji abP

baPap

baP1 11 1 )|(

1log),()(

1log),(

H(B | A)

q

i

s

j jiij

q

j

s

i jji baP

abPbp

baP1 11 1 )|(

1log),()(

1log),(

H(B)

H(A, B) =

)|()(

),(

iji

ji

abPap

baP

)|()(

),(

jij

ij

baPbp

abP

7.5

Define :

Joint Entropy

H(A | B)

H(A)

H(A, B)

H(A| B) H(B| A)

Mutual Information

The amount of information they are sharing corresponds to Information gain upon receiving bj : I(ai) − I(ai | bj) .

)()|(

log)|(

1log)(

1log);(i

ji

jiiji ap

baPbapap

baI

By symmetry:

Theorem sBaye'by );()(

)|(log);( ji

j

ijij baI

bpabP

abI

If ai and bj are independent (all noise), then P(ai , bj) = p(ai) ∙ p(bj) and hence P(ai | bj) = p(ai) I(ai ; bj) = 0. No

information gained in channel.

7.6

p(ai) P(ai | bj)

a priori a posteriori

H(A| B) H(B| A)

H(A, B)

I(A; B)

shared

joint

Average over all ai:

)(

)|(log)|();()|();(

i

jiji

ijijij ap

baPbaPbaIbaPbAI

Similarly: j j

ijiji bp

abPabPBaI

)()|(

log)|();(

);()()(

),(log),();()();( ABI

bpapbaP

baPBaIapBAIi j ji

jiji

iii

from symmetry

)(1log

)(

),()(

1log

)(

),(),(

1log),(jj

j

iji

ii

i

jji

i j jiji bp

bp

baPap

ap

baPbaP

baP

i j

jijiji bpapbaPbaPBAI )](log)(log),()[log,();(

= H(A) + H(B) − H(A, B) ≥ 0 H(A, B) H(A) + H(B)

We know H(A, B) = H(A) + H(B | A) = H(B) + H(A | B). I(A ; B) = H(A) − H(A | B) = H(B) −

H(B | A) ≥ 0

H(A | B) H(A) and H(B | A) H(B).

By Gibbs I(A; B) ≥ 0. Equality only if P(ai, bj) = p(ai)∙p(bj) [independence].

7.6