20150101c chapter 5plus6

7/25/2019 20150101c Chapter 5plus6

1/28

Weak Typicality and Strong Typicality

Yuan Luo

7/25/2019 20150101c Chapter 5plus6

2/28

Content

Ch5. Weak Typicality

5.1 The Weak AEP 5.2 The Source Coding Theorem

5.3 Efficient Source Coding

5.4 The Shannon-McMillan-Breiman Theorem

Ch6. Strong Typicality

6.1 Strong AEP 6.2 Strong Typicality Versus Weak Typicality

7/25/2019 20150101c Chapter 5plus6

3/28

5.1 The Weak AEP

We consider an information source {: 1 where are i.i.d. with distribution . We use to denotethe generic random variable and to denote thecommon entropy for all

, where

. Let

, , , . Since are i.i.d.,

7/25/2019 20150101c Chapter 5plus6

4/28

Note that the is a random variable because it isa function of the random variables , , , .We nowprove an asymptotic property of

called the weak

asymptotic equi-partition property (weak AEP).

5.1 The Weak AEP

7/25/2019 20150101c Chapter 5plus6

5/28

Since log ,log

, are i.i.d., using the Khinchine weak law of largenumbers, we have

log log

log )

in probability. So most of the sequences behave similarly in probability

| log )| ,see follows.

7/25/2019 20150101c Chapter 5plus6

6/28

Theorem(Weak APE I) If the source is over binary

field,

1 log

in probability as ,i.e., for any 0 andsufficiently large ,

Pr log 1 .

5.1 The Weak AEP

7/25/2019 20150101c Chapter 5plus6

7/28

Proposition. 1. 2. 3. 4.1.

0,

12. 0, 1 3. 0, 14. 0, 1 It is easy to see that 1. 2. 4. It is also easy to see that 1. 3. 4.As to 4. 1., 0

7/25/2019 20150101c Chapter 5plus6

8/28

Definition 5.2. If the source is over binary field,

the weakly typical set with respect to isthe set of sequences , , , such that

1 log

or equivalently,

1

log

where is an arbitrarily small positive real number.The sequence in are called weakly -typicalsequences.

5.1 The Weak AEP

7/25/2019 20150101c Chapter 5plus6

9/28

Theorem (Weak AEP II) If the source is over

binary field, any 0, If ,then

2 2 . For n sufficiently large,

Pr 1 For n sufficiently large,

12 | | 2 .

5.1 The Weak AEP

Equi-probability

Partition of the

probability space.

Tiny in the sample space and

good for compression.

7/25/2019 20150101c Chapter 5plus6

10/28

5.2 The Source Coding Theorem

A block source coding scheme with errors:

Let, , , be i.i.d. r.v. with generic. A coding schemeis:

where

is over

0,1 . . , and 0 , 1 . ..

is a special subset of such that the map is one-to-oneon , where |||| . Its easy to see that, the codingrate is

|| || ||||and the decoding error probability is .

7/25/2019 20150101c Chapter 5plus6

11/28

Shannon source coding theorem (for binary source)

Direct part

0 , a coding-decoding scheme using the formerdescription such that, if n is large enough, then


Proof.

Step1. Since the result is symmetric, but the mathematic tools AEP are notcompletely symmetric, we set a new error upper bound to replace to solvethis problem. For 1 0 , let 1 0 such that

log

=

.

7/25/2019 20150101c Chapter 5plus6

12/28

Step2. Error and Rate. In the former coding scheme, let , then

Weak AEP II: { } < < Weak AEP II: log| |

Weak AEP II: log| |

log1

log1 =

7/25/2019 20150101c Chapter 5plus6

13/28

Converse part

If there exists a former coding-decoding scheme suchthat ,then 1


Proof. Only need to prove that lim =0, is the coding set.For

, let

be the complementary set of

. If n is large,

|| max 2 2 + 2+

So lim .And then lim =0 since the leftpart has no relation with .

7/25/2019 20150101c Chapter 5plus6

14/28

Theorem

Let , , , be a random binary sequence oflength . Then with equality if and onlyif are drawn i.i.d. according to the uniformdistribution on

0 , 1


Since each symbol in is a bit and the rate of thebest possible code describing is 1 bit per, , , are called fair bits, with the connotationthat they are incompressible.

7/25/2019 20150101c Chapter 5plus6

15/28

Theorem

For a stationary source with entropy rate ,

Pr lim1 log Pr 1


7/25/2019 20150101c Chapter 5plus6

16/28

Content

Ch5. Weak Typicality

5.1 The Weak AEP 5.2 The Source Coding Theorem



Ch6. Strong Typicality

6.1 Strong AEP 6.2 Strong Typicality Versus Weak Typicality

7/25/2019 20150101c Chapter 5plus6

17/28

6.1 Strong AEP

We consider an information source {: 1 where are i.i.d. with distribution . We use to denotethe generic random variable and to denote thecommon entropy for all , where . Let , , , .

7/25/2019 20150101c Chapter 5plus6

18/28

Let ,) = 1

0 .

Then ,), ,), are i.i.d. and E,)= .Using the Khinchine weak law of large numbers, we have

, , E,)=in probability. So most of the sequences behave similarly in probability

| ; |

see follows.

7/25/2019 20150101c Chapter 5plus6

19/28

Definition 6.1. The strongly typical set withrespect to is the set of sequences , , , such that ; 0 ,and

| ; |

where ; is the number of occurrences of in thesequence x and is an arbitrarily small positivereal number. The sequences in are calledstrongly -typical sequences.

6.1 Strong AEP

7/25/2019 20150101c Chapter 5plus6

20/28

Theorem (Strong AEP)

There exists 0 such that 0 as 0, andthe following hold: If ,then

2 2 . For n sufficiently large,

Pr

1 For n sufficiently large,

12 | | 2 .

6.1 Strong AEP

7/25/2019 20150101c Chapter 5plus6

21/28

Theorem

For sufficiently large n, there exists 0 suchthatPr 2

6.1 Strong AEP

7/25/2019 20150101c Chapter 5plus6

22/28

6.2 Strong Typicality Versus Weak Typicality

Strong typicality is more powerful and flexible than

weak typicality as a tool for theorem proving for

memoryless problems, but it can be used only forrandom variables with finite alphabets.

Strong Typicality:

,

, log log

, log log

Weak Typicality: 1 log

7/25/2019 20150101c Chapter 5plus6

23/28

Proposition 6.5.

For any

x

, if

x ,then

x , where 0 as 0.

Proof. See the 1st part of Theorem (Strong AEP).

Note: Strong typicality implies weak typicality, butthe converse is not true.


7/25/2019 20150101c Chapter 5plus6

24/28

Example:

Let be distributed with such that 0 0.5;1 0.25; and 2 0.25. Consider a sequence xof length n and let be the relative frequency ofoccurrence of symbol in x, i.e., ; x, where 0,1,2.


7/25/2019 20150101c Chapter 5plus6

25/28

In order for the sequence x to be weakly typical, we

need

1 log

0 log0.5 1 log0.25 2 log0.25

0.5 log0.5 0.25 log0.25 0.25 log0.25.


7/25/2019 20150101c Chapter 5plus6

26/28

Obviously, this can be satisfied by choosing

for all .But alternatively we canchoose

. , . and

. With

such a choice of

the sequence x is weakly

typical with respect to but obviously not stronglytypical with respect to

, because the relativefrequency of occurrence of each symbol is ,which is not close to

1, 2.


7/25/2019 20150101c Chapter 5plus6

27/28

Exercises: Show examples such that

and 2

and

3 and 4 and

7/25/2019 20150101c Chapter 5plus6

28/28

Thank you!

20150101c chapter 5plus6

Documents