Codebook Organization to Enhance Maximum A

Posteriori Detection of Progressive Transmission of

Vector Quantized Images Over Noisy Channels

Ren-Yuh Wang Eve A. Riskin Richard Ladner

1

1

This work was supported by NSF Grants No. MIP-9110508 and CCR-9108314, an NSF Young

Investigator Award, and a Sloan Research Fellowship. Ren-Yuh Wang was with the Department

of Electrical Engineering, Box 352500, University of Washington, Seattle, WA 98195-2500. He is

now with FutureTel, Inc., 1092 E. Arques Avenue, Sunnyvale, CA 94086. Eve A. Riskin is with

the Department of Electrical Engineering, University of Washington, and Richard Ladner is with

the Department of Computer Science and Engineering, University of Washington. This work was

appeared in part in the 1993 Proceedings of the International Conference on Acoustics, Speech,

and Signal Processing.

Abstract

We describe a new way to organize a full search vector quantization codebook so that

images encoded with it can be sent progressively and have resilience to channel noise. The

codebook organization guarantees that the most signi�cant bits (MSB's) of the codeword

index are most important to the overall image quality and are highly correlated. Simulations

show that the e�ective channel error rates of the MSB's can be substantially lowered by

implementing a Maximum A Posteriori (MAP) detector similar to one suggested by Phamdo

and Farvardin [12]. The performance of the scheme is close to that of Pseudo-Gray [22] coding

at lower bit error rates and outperforms it at higher error rates. No extra bits are used for

channel error correction.

Contents

1 Introduction 1

2 Principal Component Partitioning 3

3 Distortion From Errors in Di�erent Codeword Index Bits 6

4 Decreasing Average Distortion by Local Switching 8

5 MAP Detection for Images 15

5.1 Applying MAP detection to images coded with organized VQ codebooks : : 15

5.2 Prediction of E�ective Error Rate : : : : : : : : : : : : : : : : : : : : : : : : 18

5.3 Fast MAP Decoding Scheme : : : : : : : : : : : : : : : : : : : : : : : : : : 21

6 Results 23

7 Conclusion And Future Work 27

8 Acknowledgements 27

1

1 Introduction

Vector quantization (VQ) [1] [5] [6] is a lossy compression technique that has been used

extensively for image compression. Full search VQ leads to higher image quality than is

given by the popular tree-structured VQ (TSVQ), but TSVQ is amenable to progressive

transmission due to its built-in successive approximation character. In a progressive image

transmission system [18], the decoder reconstructs increasingly better reproductions of the

transmitted image as bits arrive to allow for early recognition of the image. In [18], Tzou

describes TSVQ's suitability for progressive transmission. Due to the successive approxi-

mation nature of TSVQ, the farther down the tree the vector is encoded, the better the

reproduction. In [16], we organized a full search VQ so that images coded with it could be

sent progressively as in TSVQ. Related work on ordered VQ codebooks has been studied by

Poggi and Cammorota to reduce the bit rate of standard and progressive transmission VQ

[2, 14], by Nasrabadi and Feng to lower the bit rate in �nite-state VQ [10] and in our earlier

work [15].

One problem of transmitting any VQ codeword index is that VQ is sensitive to channel

errors. When VQ codeword indexes are transmitted over a noisy channel, channel errors can

cause a signi�cant increase in distortion if the VQ codebook is not organized for resilience

to channel noise.

There has been signi�cant previous research on the transmission of VQ indexes over noisy

channels. Phamdo, Farvardin and Moriya [13] designed a channel-matched TSVQ scheme

that does not need extra bits for error correction. They developed an iterative algorithm

to design a set of codebooks to minimize a modi�ed distortion for di�erent channel error

rates. Their experiments showed substantial improvements over ordinary TSVQ when the

channel is very noisy, but because their scheme uses a di�erent codebook for each channel

error rate, if there is a channel mismatch, the encoder does not use the best codebook for the

channel. Phamdo and Farvardin also implemented Maximum A Posteriori (MAP) detection

1

of Markov sources in which the correlation between successive VQ indexes is used to correct

channel errors [12].

In other work, Sayood and Borkenhagen [17] used residual redundancy to correct channel

errors in the design of a joint source-channel DPCM image coding system. Zeger and Gersho

[22] developed Pseudo-Gray coding to reassign the indexes of a VQ codebook to reduce the

distortion introduced by noisy channels. They rearrange the codebook so that codewords

with similar indexes lie near each other in the input space. They iteratively switch the

codewords when the switch lowers the distortion caused by channel noise. They obtain a

reordered codebook with distortion at a local minimum. Cheng and Kingsbury designed

robust VQs using a codebook organization technique based on minimum cost perfect match-

ing [3] and Knagenhjelm used a Kohonen method for designing VQs robust to channel noise

[8]. Hung and Meng [7] adaptively change the codebook at the receiver, depending on the

channel bit error rate, without su�ering performance degradation for noiseless transmission.

They also developed a modi�ed annealing method to generate VQ codebooks that improve

channel error robustness and have little performance degradation for noiseless transmission.

In this paper we extend our methods for using an organized �xed rate full search codebook

for progressive transmission [16] to protect against channel errors. First, a full search progres-

sive transmission tree is constructed using the method of Principal Component Partitioning

(PCP) [16]. The result is an assignment of codeword indexes which allows for progressive

transmission and at the same time gives protection against channel noise. Second, the full

search progressive transmission tree is modi�ed by switching nodes in a way reminiscent of

the switching that is done in Pseudo-Gray Coding [22]. The switching improves the ability

to protect against channel noise without a�ecting the progressive transmission performance

over noiseless channels. Finally, because the PCP method causes the most signi�cant bits

of the codeword indexes to become highly correlated, we use a MAP detection scheme to

further decrease distortion for noisy channels.

The remainder of the paper is organized as follows. In Section 2, we describe Principal

2

Component Partitioning (PCP), our codebook organization technique. A complete descrip-

tion of it can be found in [16]. In Section 3, we derive an equation which can be used to

calculate the distortion introduced by errors in each bit position of the codeword indexes

and describe our switching algorithm in Section 4. In Section 5, we apply the MAP de-

tector to correct channel errors in progressive image transmission, predict the e�ective and

critical channel error rates of our MAP scheme, and propose a fast decoding scheme for the

MAP detector. In Section 6, we present the results of using our MAP detector on images

transmitted over simulated noisy channels. Finally, we conclude in Section 7.

2 Principal Component Partitioning

We developed Principal Component Partitioning (PCP) in [16] to build a full search progres-

sive transmission tree to organize a full search VQ codebook for progressive transmission.

The tree gives full search VQ the successive approximation character that is built into TSVQ.

The progressive transmission tree is a balanced tree whose terminal nodes or leaves are la-

beled by VQ codewords and whose internal nodes are labeled by intermediate codewords

derived from the leaf codewords. The tree is used to reassign the original codeword indexes

to new indexes that can be used for progressive transmission.

To build the tree, we initially �nd a hyperplane perpendicular to the �rst principal

component of the training set used to design the full search codebook. The hyperplane

partitions the codewords into two equal size sets. An iterative process is used to adjust the

hyperplane [16] and the iteration terminates with two equal size sets of codewords which

are used to build the next layer of the tree. The recursive application of PCP leads to a

top-down construction of the full search progressive transmission tree. Wu and Zhang also

used a method of principal components to build TSVQ's [21].

Fig. 1 (a) is a 256�256 magnetic resonance (MR) image displayed at 8 bits per pixel (bpp).

The image is coded to 2 bpp in Fig. 1 (b) with a size 256 codebook with vector dimension

3

Figure 1: (a) The original 256 � 256 magnetic resonance image at 8 bpp (left) and (b) the

coded image at 2 bpp with a GLA VQ codebook of size 256 and vector dimension 4 (right).

4

Figure 2: The intermediate images from 1 (top left) to 8 bpv (bottom right) coded by a 4

dimensional codebook of size 256 organized by PCP.

5

4 designed on a training set of 20 MR images with the generalized Lloyd algorithm (GLA)

[9]. We will use this MR image codebook throughout the paper. Fig. 2 is an example of

the quality of intermediate progressive transmission images resulting from organizing the

codebook with PCP. These images range from 1 bit per vector (bpv) to 8 bpv.

As a by-product of the PCP codebook organization, we have found that the PCP code-

book indexes have natural resilience to channel noise. This is because the codewords whose

indexes di�er only in the least signi�cant bits (LSB's) lie near each other in the full search

progressive transmission tree. Thus, errors occurring in the LSB's of the codeword index

produce little distortion. We shall examine this further in Section 3 and see that errors in

di�ering bits of the codeword index have di�erent e�ects on the �nal image. We shall see in

Section 5 that they can be protected to a varying extent by a MAP detector.

3 Distortion From Errors in Di�erent Codeword Index

Bits

In this section, we derive an equation for calculating the distortions that are introduced by

errors in each bit position of the codeword index. The equation is then used to calculate the

distortion that is introduced by channel noise for our MR image codebook.

Given a size N codebook fC

0

, C

1

,: : : ,C

N�1

g designed on a training set T , we de�ne T

i

as the subset of T whose nearest codeword is C

i

, the centroid of T

i

. We de�ne W

i

as the

number of vectors in T

i

and D

i

as the total distortion measured between T

i

and C

i

. When

there are channel errors such that every input in T

i

is mapped to C

j

, we incur an increase

in the total distortion of amount (D

jji

�D

i

) where

D

jji

=

X

X2T

i

jjX � C

j

jj

2

=

X

X2T

i

jj(X � C

i

) + (C

i

� C

j

)jj

2

= D

i

+W

i

jjC

i

� C

j

jj

2

(1)

where jjC

i

�C

j

jj

2

is the squared distance betweenC

i

and C

j

. We de�ne �

i

to be the probability

that the i-th bit (�

8

= most signi�cant bit (MSB), �

1

= least signi�cant bit (LSB)) of the

6

codeword index is in error. We use the binary symmetric channel (BSC) and in the analysis

that follows, we assume that at most one bit of a codeword index is in error at a time. The

average distortion of a vector transmitted over a BSC channel is then

D

av

=

1

W

N�1

X

i=0

N�1

X

j=0

Q

jji

D

jji

(2)

where W =

P

N�1

j=0

W

j

is the size of the training set. The quantity Q

jji

is the conditional

probability that the index of C

j

is received given that the index of C

i

was transmitted. By

Equation 1,

D

av

=

1

W

N�1

X

i=0

N�1

X

j=0

Q

jji

(D

i

+W

i

jjC

i

� C

j

jj

2

);

and

D

av

=

1

W

N�1

X

i=0

D

i

N�1

X

j=0

Q

jji

+

1

W

N�1

X

i=0

N�1

X

j=0

Q

jji

W

i

jjC

i

� C

j

jj

2

(3)

Equation 3 is the same as Equation 7 in Farvardin [4]. Because

P

N�1

j=0

Q

jji

= 1, the �rst term

of Equation 3 is exactly the average distortion when the codebook is used with a noiseless

channel. We de�ne this as D =

P

N�1

i=0

D

i

W

. For i

k

, which we de�ne to di�er from index i only

in the k-th bit,

Q

i

k

ji

= Q

�

k

1� �

k

; (4)

where Q =

Q

log

2

N

j=1

(1 � �

j

) is the probability that a codeword index is transmitted with no

error. Since we assumed the probability of more than one error per codeword to be highly

unlikely, we get

N�1

X

j=0;j 6=i

Q

jji

'

log

2

N

X

k=1

Q

i

k

ji

(5)

Equation 3 then reduces to

D

av

' D +

Q

W

log

2

N

X

k=1

�

k

1� �

k

N�1

X

i=0

W

i

jjC

i

� C

i

k

jj

2

(6)

We organized the MR codebook from Section 2 with the PCP algorithm on the same MR

training set. For this codebook and data set, we calculate:

D

av

' D + A(135

�

8

1� �

8

+ 57

�

7

1 � �

7

+ 16

�

6

1� �

6

+ 5:6

�

5

1� �

5

7

+3:8

�

4

1� �

4

+ 3:1

�

3

1� �

3

+ 1:6

�

2

1� �

2

+

�

1

1� �

1

) (7)

where

A =

Q

W

N�1

X

i=0

W

i

jjC

i

� C

i

1

jj

2

(8)

is normalized against the bit sensitivity of the LSB. Let us de�ne the bit sensitivity for the

k-th LSB to be

S

k

=

N�1

X

i=0

W

i

jjC

i

� C

i

k

jj

2

(9)

Equation 7 shows that for our data set,

S

8

= 135S

1

:

This means that an error in the MSB will result in an increase in distortion that is 135 times

larger than that caused by an error in the LSB (if the channel error rates of the MSB and

LSB are the same). Fig. 3 shows the MR images at 2 bpp with 50% errors in each bit of the

codeword index. We can clearly see that errors occurring in the MSB of the codeword index

(top left) would make the images look much worse than errors in the LSB (bottom right).

Thus, the MSB's of the codeword index must be carefully protected against channel noise.

Generally speaking, using the PCP method to construct the full search progressive trans-

mission tree will always have the e�ect that errors in the MSB's will cause more distortion

than errors in the LSB's. Due to the PCP, the MSB separates the codewords into two equal

size sets which are fairly well separated. Thus, an error in the MSB will cause the decoded

codeword to be far from the one that was transmitted. Each succeeding split separates

codewords that are closer together. Thus channel errors occurring in the LSB's are less

detrimental to the image quality than are errors occurring in the MSB's.

4 Decreasing Average Distortion by Local Switching

After a codebook is organized by PCP, the average distortion between the original image

and the decoded image transmitted over noisy channels is expressed as Equation 6. The �rst

8

Figure 3: The coded images at 2 bpp with 50% errors occurring in the codeword index from

the MSB (top left) to LSB (bottom right) (codebook size 256).

9

z

z

vC

01

C

23

vv

z

z

E

E

E

E

E

E

E

E

E

E

E

C

2

C

3

C

0

C

1

C

0123

Figure 4: An example of a size 4 codebook organized by PCP and the rate 0 (fC

0123

g) and

rate 1 (fC

01

; C

23

g) intermediate codebooks.

term on the right hand side of Equation 6 is the average distortion for a noiseless channel

and is �xed for a given codebook. In this section, we develop a method of switching nodes

in the full search progressive transmissions tree (related to Pseudo-Gray Coding [22]) which

further decreases the second term of Equation 6, the distortion due to channel errors. The

switching method does not a�ect the progressive transmission performance obtained from

the PCP alone over noiseless channels.

We describe the switching method on a simple example of the size four codebook in

Fig. 4. The four codewords fC

0

; C

1

; C

2

; C

3

g are organized by the PCP and a progressive

transmission tree is �t to the codebook to obtain intermediate codewords fC

01

; C

23

g and

fC

0123

g. Two possible forms of arranging the codebook are shown in Fig. 5(a) and Fig. 5(b).

We wish to select the codebook arrangement with the smaller D

av

in Equation 6. That is,

we compare the values of

(W

0

+W

2

)jjC

0

� C

2

jj

2

+ (W

1

+W

3

)jjC

1

� C

3

jj

2

and

(W

0

+W

3

)jjC

0

� C

3

jj

2

+ (W

1

+W

2

)jjC

1

� C

2

jj

2

:

The codebook arrangement in Fig. 5(b) is obtained by just switching C

2

and C

3

in Fig. 5(a);

this clearly does not change the intermediate codeword C

23

. Similarly, if we were to switch

10

C

0123

�

�

�

�

��

H

H

H

H

Hj

C

01

�

J

J

J

J

J

C

23

�

J

J

J

J

J

C

0

C

1

C

2

C

3

0 1

0 1 0 1

(a)

C

0123

�

�

�

�

��

H

H

H

H

Hj

C

01

�

J

J

J

J

J

C

23

�

J

J

J

J

J

C

0

C

1

C

3

C

2

0 1

0 1 0 1

(b)

Figure 5: Two possible codebook organizations by PCP.

codewords C

0

and C

1

, the intermediate codeword C

01

would not change. Thus, the progres-

sive transmission performance over noiseless channels is unchanged by this local switching

(LS).

In the general case, there are N codewords organized as leaves of a full search progressive

transmission tree generated by the PCP. By a local switch, we mean the exchange of a left

subtree with a right subtree as described in Fig. 6. A local switch has the e�ect that for some

i and x where 1 � i � log

2

N and 0 � x < 2

log

2

N�i

, and for all y, 0 � y < 2

i�1

, the codewords

with indexes y + 2

i

x and y + 2

i�1

+ 2

i

x are switched. We call i the level of the local switch.

The root is at level log

2

N and the leaves at level 0. By switching nodes in the full search

progressive transmission tree in this manner, it is possible to further reduce D

av

. Indeed,

there is an optimal set of local switches which minimizes the average distortion D

av

but

�nding the optimal set of local switches requires searching exponentially many possibilities.

We are left to explore alternative heuristics that lead to suboptimal solutions.

11

Recall from Section 3 that the bit sensitivity of the k-th bit is de�ned to be

S

k

=

N�1

X

i=0

W

i

jjC

i

� C

i

k

jj

2

:

If we assume that the transmission error rate in all bit positions is identical, minimizing D

av

is equivalent to minimizing

P

log

2

N

k=1

S

k

according to Equation 6 and Equation 9. Thus, the

goal of our heuristics is to minimize this sum. As a practical matter, a local switch at level

log

2

N does not change the value of

P

log

2

N

k=1

S

k

. More generally, a local switch at level i does

not change the value of

P

log

2

N

k=i+1

S

k

. Thus, to test whether a local switch at level i decreases

the average distortion, D

av

, it is only necessary to test whether it decreases the value of

P

i

k=1

S

k

.

There are a number of heuristics to apply LS to decrease D

av

. Any heuristic we describe

starts with a full search progressive transmission tree produced by the PCP as the initial

tree. Listed below are four possibilities.

Simple Greedy Heuristic: In this approach, simply search the full search progressive

transmission tree for the local switch which produces the largest decrease in D

av

. Apply this

local switch and repeat the process until no local switch reduces D

av

.

Simple Greedy Heuristic with Simulated Annealing: In this approach, apply ran-

dom switches to the initial tree without regard to decreasing D

av

(this is the annealing step).

With the result of this annealing step, apply the simple greedy heuristic. Repeat this algo-

rithm a number of times with di�erent starting trees and choose the tree with the smallest

D

av

. Our experience is that the initial tree is a good starting point for the simple greedy

heuristic, so that the annealing step should only be performed on a small percentage of the

nodes. A good choice appears to be choosing to randomly switch about 5% of the nodes

of the initial tree. The number of times the annealing should be repeated depends on the

number of nodes in the initial tree. For our trees with 256 leaves, 100 annealing trials were

adequate.

12

A B B A

0 10

Local Switch at Node X

1

Level i Level i XX

Figure 6: A local switch at a node switches its left and right subtrees.

Top Down Greedy Heuristic: In this approach, simply search the tree level by level

starting with the root, to �nd any local switch which decreases D

av

. Repeat this process

until no local switch decreases D

av

. Starting the search at the top of the tree has the

advantage that usually, local switches nearer the root cause larger decreases in D

av

than

switches lower down the tree, because more codewords are a�ected by the switch.

Top Down Greedy Heuristic with Simulated Annealing: This approach is similar

to the Simple Greedy Heuristic with Simulated Annealing method, except that the top down

greedy heuristic is used in place of the simple greedy heuristic.

All these heuristics and most of the others we have tried give very similar results. The

simple greedy and top down greedy heuristic take about the same number of iterations before

convergence. Adding simulated annealing increases the execution time by a large factor with

only small decreases in the average distortion.

To obtain the best result, we used the Top Down Greedy Heuristic with Simulated An-

nealing method to further organize our MR codebook from Section 2. We calculate the new

13

average distortion over noisy channels by Equation 6:

D

av

' D + A(123

�

8

1� �

8

+ 48

�

7

1 � �

7

+ 13

�

6

1� �

6

+ 5:0

�

5

1� �

5

+2:9

�

4

1� �

4

+ 2:7

�

3

1� �

3

+ 1:6

�

2

1� �

2

+

�

1

1� �

1

) (10)

Comparing Equation 10 and Equation 7, we see that errors introduce less distortion for our

codebook once it has been further organized by the LS algorithm. If the error rates are the

same on each bit, that is,

�

i

= �; i = 1; 2; : : : ; 8;

Equation 7 becomes

D

av

' D + A

223�

1� �

and Equation 10 becomes

D

av

' D + A

197�

1� �

:

We can see that the LS algorithm decreases the increase in distortion caused by channel

errors by 11.6% in this case.

As we de�ned in Section 3, the bit sensitivity S

k

is the amount of increased distortion

caused by the channel errors occurring in the k-th LSB of the codeword index. For our

medical image codebook, the S

k

's for a codebook organized with the PCP and the LS

algorithm are in Equation 10. As we can see, the S

k

's are monotonically decreasing with

k in a roughly exponential manner. This shows that the MSB's, whose bit sensitivities are

larger, are more important to the image quality than the LSB's. While this monotonic e�ect

is of course dependent on the data set and can not be proved in general for real images,

we prove a precise exponential relationship among the bit sensitivities for a 2-dimensional

lattice VQ [19].

14

5 MAP Detection for Images

In Section 3, we showed that the MSB's of codeword indexes from a codebook organized by

the PCP are much more important to the image quality than the LSB's. Another conse-

quence of the full search progressive transmission tree is that it results in a high amount of

correlation between the MSB's of the codeword indexes. This is because images vary slowly

and neighboring input vectors are likely to be coded from the same region of the organized

codebook. This is the same observation used by Neuho� and Moayeri in their interblock

noiseless coding for TSVQ [11].

Here we illustrate the high correlation between MSB's of the codeword indexes from

our organized VQ. Fig. 7 is a display of the MSB (top left) to the LSB (bottom right) of

the codeword indexes for a magnetic resonance brain image coded with our organized VQ

(0 = black and 1 = white). The MSB is obviously highly correlated and is even a coarse

approximation to the original image. An image coded with an unorganized codebook is

not likely to display such high correlation. We see from Fig. 7 that the LSB is much more

random but fortunately, this bit is not important to the image quality.

In this section, we use this correlation in the MSB's of the codeword index to apply MAP

detection to images with channel errors. In addition, we predict the e�ective error rate,

describe the fast MAP decoding scheme, and predict the critical channel error rate based on

training data.

5.1 Applying MAP detection to images coded with organized VQ

codebooks

We send one bit plane of the codeword index at a time from the MSB to the LSB. Because

the MSB's of the codeword indexes of an image are highly correlated, we can use the re-

dundancy between them to correct channel errors for progressive transmission over noisy

communication channels. To implement this, we use a variation of Phamdo and Farvardin's

15

Figure 7: MSB's (top left) to LSB's (bottom right) of image coded with organized full search

codebook of size 256 (0 = black, 1 = white). Notice that the MSB's are much more highly

correlated than the LSB's.

16

-

T

1

T

2

T

3

T

8

T

0

T

4

T

7

T

6

T

5

R

1

R

2

R

3

R

8

R

0

R

4

R

7

R

6

R

5

Figure 8: T

i

and R

i

are the transmitted and received bits over a BSC.

MAP detector [12]. The received bit plane is simply scanned with a sliding 3� 3 bit block

and the middle bit is either changed or unchanged based on the channel bit error rate (BER)

and conditional probabilities calculated from training data of the received 3� 3 bit block.

In Fig. 8, T

0

and its eight neighboring bits T

i

, 1 � i � 8 are a block of bits from one bit

plane of the codeword index. They are transmitted over a noisy BSC and the corresponding

received bit of T

i

is the random variable R

i

which may be ipped by the channel noise. It

is also helpful to recognize that T

i

is also a random variable, namely, that the probability of

the bit T

i

, 0 � i � 8 is statistically determined by the input images. Let r

i

, 0 � i � 8, be a

�xed set of 9 received bits. Given knowledge of the error rate and the training set, if

Pr(T

0

= r

0

jR

i

= r

i

; 0 � i � 8) � Pr(T

0

= �r

0

jR

i

= r

i

; 0 � i � 8); (11)

the received bit r

0

is decoded to �r

0

; otherwise it is unchanged. In other words, we change

the central bit of the 3 � 3 block if the probability that the transmitted central bit is the

same as the received central bit, conditioned on the received 3� 3 block, is less than 0.5.

The e�ect of the error rate on the probabilities in Equation 11 can be factored out so

that the probabilities (which we estimate as relative frequencies) depend only on the training

set. The resulting inequality has many terms but, if the error rate � is much less than 1, the

following inequality (where the symbol ^ represents a logical AND) can be used in its place

with negligible loss in performance

Pr(T

i

= r

i

; 0 � i � 8)(1� �)

9

+

17

8

X

j=1

Pr(T

j

= �r

j

^ T

i

= r

i

; i 6= j; 0 � i � 8)�(1� �)

8

�

Pr(T

0

= �r

0

^ T

i

= r

i

; 1 � i � 8)�(1� �)

8

+

8

X

j=1

Pr(T

0

= �r

0

^ T

j

= �r

j

^ T

i

= r

i

; i 6= j; 1 � i � 8)�

2

(1� �)

7

(12)

The probabilities in Equation 12 depend only on the training data. As examples, Pr(T

i

=

r

i

; 0 � i � 8) is simply the percentage of 3�3 blocks from the training set which are identical

to the block which contains exactly r

0

; r

1

; :::; r

8

, and Pr(T

j

= �r

j

^ T

i

= r

i

; i 6= j; 0 � i � 8)

is the percentage of 3 � 3 blocks from the training set which are identical to the block

which contains r

0

; :::; r

j�1

; �r

j

; r

j+1

; :::; r

8

. The �rst term on the left side in Equation 12 is the

probability of transmitting T

i

, 0 � i � 8 without channel errors. The second term on the

left side is for when T

0

is transmitted correctly but there is one error in T

i

; 1 � i � 8. In the

�rst term on the right side, T

0

is transmitted incorrectly but all other bits are transmitted

correctly. Finally, the second term on the right side is due to transmitting T

0

incorrectly along

with one additional error in T

i

, 0 � i � 8. Our simulation results show that Equation 12 is

a good approximate decision criterion. Adding more terms to allow for more channel errors

requires higher decoding complexity and gave little performance improvement.

5.2 Prediction of E�ective Error Rate

The MAP decoder changes or does not change the received data based on Equation 12. Thus,

it not only corrects errors but may also introduce errors. As a result, the e�ective error rate

of our MAP scheme consists of two parts. One is due to the MAP decoder not correcting

an error caused by channel noise and the other is due to the MAP decoder introducing

errors. When we design the MAP detector, we can calculate these two probabilities based

on the training set and predict the e�ective error rate in advance. Our results show that the

predicted e�ective error rates are very close to the simulated values.

For a bit T

0

transmitted over a BSC with channel error rate �

channel

and decoded with

18

Noisy Channel

�

channel

- MAP

Decoder

-

T

0

-

R

0

D

0

Figure 9: Bit T

0

is transmitted over noisy channels and received as R

0

and then decoded as

D

0

.

our MAP scheme in Fig. 9, the received bit is R

0

, and R

0

is decoded to D

0

by the MAP

scheme. Because the e�ective error rate �

e�ective

is comprised of two terms, which depend

only on �

channel

and the training set, it can be calculated ahead of time as follows:

�

e�ective

= Pr(D

0

=

�

T

0

) = Pr(R

0

= T

0

^D

0

=

�

R

0

) + Pr(R

0

=

�

T

0

^D

0

= R

0

) (13)

= (1� �

channel

)Pr(D

0

=

�

R

0

jR

0

= T

0

) + �

channel

Pr(D

0

= R

0

jR

0

=

�

T

0

):

For a �xed 3� 3 vector r

0

; r

1

; :::; r

8

of bits, we de�ne the integer

r =

8

X

i=0

r

i

2

i

to simply be the natural interpretation of the 3 � 3 binary vector as an integer. We also

de�ne r(j

1

; j

2

; : : : ; j

n

) to di�er from r in the j

1

-th, j

2

-th,: : :, and j

n

-th bits. Finally, we de�ne

the indicator function

I

frg

=

8

>

<

>

:

1; if r

0

is changed by the MAP detector due to r being received

0; otherwise.

For r = 0; 1; : : : ; 511, de�ne Pr[r] = Pr(T

i

= r

i

; 0 � i � 8) to be the a priori probability of

transmitting 3 � 3 binary block r. By considering the probabilities of all 512 patterns and

their probabilities of being changed by the MAP detector, we get the following equation

Pr(D

0

=

�

R

0

jR

0

= T

0

) = Pr(R

0

is changed by MAPjR

0

= T

0

)

19

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

10-5 10-4 10-3 10-2 10-1 100

o

o

ooo

o

o

oo

o

o

o

oooooo

x

x

xxx

x

x

x

x

x

xxxxxxxx

*

*

***

*

*

***********

++++++++++++++++++

channel error rate

deco

ded

erro

r ra

te /

chan

nel e

rror

rat

e

Solid Line = Simulated Value

o = The First Bit (MSB)

x = The Second Bit

* = The Third Bit

+ = The Fourth Bit

Figure 10: Simulated and estimated e�ective error rates over di�erent channel error rates

for Magnetic Resonance training data.

=

511

X

r=0

Pr[r]

�

q

8

I

frg

+

P

1�j

1

�8

�q

7

I

fr(j

1

)g

+

P

1�j

1

<j

2

�8

�

2

q

6

I

fr(j

1

;j

2

)g

+ : : :+ �

8

I

fr(j

1

;j

2

;:::;j

8

)g

�

where q = 1� �. The �rst term on the right side of the above equation is the case in which

R

1

to R

8

are all received correctly, and the second term is for when one bit is in error. The

following terms are for when two through eight neighboring received bits are in error.

The term Pr(D

0

= R

0

jR

0

=

�

T

0

) in Equation 13 can be also obtained in the same manner

and thus we can directly calculate �

e�ective

. The calculated and simulated results for the �rst

4 MSB's from our medical image codebook are shown in Fig. 10. We see that the curves

estimated from the training data �t the simulated results very closely. This result means

that we can accurately predict the e�ective channel error rate from the training set and the

channel error rate. We point out that the MAP detector works slightly worse here than

on the test set; the reason is that two images from the test set of MR images include a

signi�cantly larger amount of black background than is included in the training set images.

Errors in the black background are of course more easily corrected by the MAP detector

than errors in other parts of the image.

20

5.3 Fast MAP Decoding Scheme

In this section, we show that each 3 � 3 pattern r has a channel error rate �

min

(r) below

which the MAP decoder would not change the central bit of the block. For each r, if we

know the channel error rate is below �

min

(r), then we would not apply the MAP decoder if

pattern r is received. The estimate of the channel error rate can be derived from the signal

power received at the decoder, the signal-to-noise ratio, or by using speci�c error correcting

codes [7]. Here we show how to calculate �

min

(r) from the training set.

Equation 12, the decision equation, is quadratic in � and therefore can be expressed as:

P

1

(1� �)

2

+ P

2

�(1� �) � P

3

�(1� �) + P

4

�

2

; (14)

where the terms P

1

; P

2

; P

3

, and P

4

, whose meaning are clear from Equation 12, can be

calculated from the training set for each speci�c 3 � 3 binary block r. Then for each r,

�

min

(r) satis�es Equation 14 with equality. Thus, if the channel error rate is greater than

�

min

(r), the central bit of block r would be ipped, and if it is below �

min

(r), it would not be

changed. This simple decoding of our MAP scheme makes the MAP detection even faster.

If the channel error rate is less than �

min

(r) for all r, the MAP decoder will make no

corrections at all for any data pattern. Thus, we de�ne the critical channel error rate [12]

as:

�

critical

= min

r

�

min

(r):

Table 1 is �

critical

for the 8 bit plane images. From the table, the MSB images, which have

higher correlation, have lower values of �

critical

than the LSB images (except for the LSB).

This means that the LSB's could be corrected only if the channel is very noisy. Fortunately,

the LSB's are relatively unimportant so that the failure to correct errors in the LSB's does

not severely a�ect the image quality.

21

Bit-Image �

critical

MSB 0:001517

2nd MSB 0:003091

3rd MSB 0:009511

4th MSB 0:046284

5th MSB 0:103989

6th MSB 0:122357

7th MSB 0:197906

LSB 0:165561

Table 1: Critical error rates for the eight bit planes.

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

10-5 10-4 10-3 10-2 10-1 100

o

o

ooo

oo

oo

o

o

o

oooooo

x

xxx

xxx

xx

x

x

xxxxxxx

**

**

***

*

**********

++

+++

+++++++++++++

channel error rate

deco

ded

erro

r ra

te /

chan

nel e

rror

rat

e

o = The First Bit (MSB)

x = The Second Bit

* = The Third Bit

+ = The Fourth Bit

Figure 11: The ratio of decoded error rates over channel error rates for MAP detection on

the �rst four MSB's for the Magnetic Resonance test data.

22

6 Results

In this section, we apply our codebook organization techniques (PCP and LS which we refer

to as PCP-LS) and MAP detector to the magnetic resonance VQ codebook. In Fig. 11, we

plot the average decoded error rate against the channel error rate for the four MSB's of the

codeword index. This is for 50 simulations of a BSC and is for a set of 5 MR images not

included in the training set used to design the codebook and calculate the decision statistics.

At 1% channel bit error rate, we correct 67% (correct 75% and introduce 8%) of the errors

in the MSB, correct 43% (correct 56% and introduce 13%) of the errors in the second MSB,

correct 27% (correct 46% and introduce 19%) of the errors in the third MSB, and do not

correct any errors in the fourth MSB. Having di�erent error rates on each bit of the codeword

index is related to work done by Farvardin [4] where he obtained unequal error probabilities

through the use of channel coding and modi�ed his VQ design accordingly.

Simulations of progressive image transmission with a BER of 10% without and with

MAP detection are shown in Fig. 12 and Fig. 13. We have selected a BER of 10% to better

illustrate the e�ects of channel errors and correction in the photographic reproductions.

Here, the MAP detector is only applied to the �rst three MSB's. The image in Fig. 12 (a) is

the decoded image from only the MSB of the codeword index with 10% errors. The images

in Fig. 12 (b) to (h) are the decoded image from two MSB's to eight bits of the codeword

index with 10% errors. The image in Fig. 13 (a) is the decoded image from only the MSB of

the codeword index with 10% errors and corrected by the MAP decoder. We can see that

the MAP decoder corrects most of the errors in the background and some of the errors in

the head region. In the same manner, the images in Fig. 13 (b) and (c) are the decoded

image from two and three MSB's of the codeword index with 10% errors and corrected by

the MAP decoder. The images in Fig. 13 (d) through (h) are for four through eight bit

planes received (the MAP detector is not applied to these bits). We use the signal-to-noise

23

Figure 12: Progressive transmission of a medical image from 1 bpv to 8 bpv with 10% BER

without MAP detection.

ratio (SNR) to measure the performance of the various schemes, where the SNR in dB is:

SNR = 10 log

10

EX

2

D

:

EX

2

is the average pixel intensity of the image and D is the mean-squared-error. The signal-

to-noise ratio (SNR) is 5.13 dB for the �nal image without MAP detection and 9.55 dB for

the �nal image with MAP detection.

Fig. 14 compares the SNRs for simulated transmission of the test set over noisy channels

using di�erent error protection schemes. These are the GLA followed by PCP-LS codebook

organization without and with the MAP detector (GLA/PCP-LS and GLA/PCP-LS+MAP),

Pseudo-Gray coding [22], and a codebook that is randomly ordered. Pseudo-Gray coding

slightly outperforms the GLA/PCP-LS. One di�erence between the two schemes is that

Pseudo-Gray coding is a�ected somewhat evenly by errors in each codeword index bit, while

24

Figure 13: Progressive transmission of a medical image from 1 bpv to 8 bpv with 10% BER

with MAP detection.

25

-5

0

5

10

15

20

25

10-6 10-5 10-4 10-3 10-2 10-1 100

o

o

oo

o

o

o

o

o

o

o

o

oooooooooooooooo

xx

xx

x

x

x

xx

x

x

x

xx

x

xx

xxxxxxxxxxx

channel error rate

SNR

x = Randomly Ordered Codebook

Dash Line = GLA/PCP-LS+MAP

o = PSEUDO-GRAY

Solid Line = GLA/PCP-LS

Figure 14: SNR of the decoded images over noisy channels for di�erent error protection

schemes.

the distortions caused by errors in the PCP-LS codeword index range from large for the

MSB's to small for the LSB's. The slight di�erence in SNR's is likely attributed to the

constraint of having a tree structure for the PCP-LS whereas the Pseudo-Gray codebook

is unconstrained. However, Pseudo-Gray coding requires about 120 times the amount of

time required by the PCP-LS to organize the same codebook. The GLA/PCP-LS+MAP

scheme outperforms Pseudo-Gray coding for channel BERs higher than 0.1 % with 4.06 dB

improvement at a 10% error rate. For error rates less than 0.1 %, Pseudo-Gray coding has a

slightly higher SNR. Pseudo-Gray coding does not require any computations by the decoder

while the GLA/PCP-LS+MAP requires the MAP detector.

In Fig. 14, we also see that the GLA/PCP-LS+MAP and GLA/PCP-LS converge to the

same curve at error rates less than 10

�3

, because the MAP detector does not work at low

error rates. We show the performance of Pseudo-Gray Coding, the GLA/PCP-LS, and the

randomly ordered codebook for channel error rates of less than 10

�3

in Fig. 15 in a larger

scale. We see that because the PCP-LS also has resilience to channel noise, the performance

is much better than the randomly ordered codebook at low error rates and very close to

that of Pseudo-Gray coding. The largest di�erence between Pseudo-Gray Coding and the

GLA/PCP-LS is 0.36 dB at channel error rate 10

�3

.

26

10-6

10-5

10-4

10-3

16

17

18

19

20

21

22

23

24

25

channel error rate

SN

R

o = PSEUDO-GRAY

Solid Line = GLA/PCP

x = Randomly Ordered Codebook

Figure 15: SNR of the decoded images over noisy channels for di�erent error protection

schemes for channels of low error rates at which MAP does not work.

7 Conclusion And Future Work

We organized a full search VQ codebook for progressive transmission. We found that the

MSB's of the codeword index were most important to the image quality and were also highly

correlated. We designed a MAP scheme to correct channel errors in these bits. Simulations

show that we can get performance close to Pseudo-Gray Coding at low channel error rates

and can outperform it at higher error rates while being able to send images progressively.

Future work will include improving the MAP detector, adding channel coding to the system

along the lines of the work of Farvardin [4], improving the performance at lower channel

error rates by combining Pseudo-Gray Coding with PCP-LS+MAP, determining bounds on

the amount of distortion caused by errors in the di�erent codeword index bits, and applying

PCP-LS to other VQ problems [20].

8 Acknowledgements

Wewould like to thank the anonymous reviews for their excellent comments on the manuscript

and Professor Ken Zeger for the Pseudo-Gray code. We are particularly grateful to B. S.

Srinivas for help with the �nal simulations.

27

References

[1] H. Abut, editor. Vector Quantization. IEEE Press, Piscataway, NJ, May 1990.

[2] E. Cammarota and G. Poggi. Address vector quantization with topology-preserving

codebook ordering. In Proc. 13th GRETSI Symposium, pages 853{856, September 1991.

[3] N.-T. Cheng and N. K. Kingsbury. Robust zero-redundancy vector quantization for

noisy channels. In Proceedings International Conference on Communications, pages

1338 {1342, 1989.

[4] N. Farvardin. A study of vector quantization for noisy channels. IEEE Transactions on

Information Theory, 36(4):799{809, July 1990.

[5] A. Gersho and R. M. Gray. Vector Quantization and Signal Compression. Kluwer

Academic Publishers, Boston, 1992.

[6] R. M. Gray. Vector quantization. IEEE ASSP Magazine, 1:4{29, April 1984.

[7] A. Hung and T. Meng. Adaptive channel optimization of vector quantized data. In

Proceedings Data Compression Conference, pages 282{291, March 1993.

[8] P. Knagenhjelm. A recursive design method for robust vector quantization. In Proceed-

ings International Conference on Signal Processing Applications and Technology, pages

948{954, Boston, MA, November 1992.

[9] Y. Linde, A. Buzo, and R. M. Gray. An algorithm for vector quantizer design. IEEE

Transactions on Communications, 28:84{95, January 1980.

[10] N. M. Nasrabadi and Y. Feng. Vector quantization of images based upon the Koho-

nen self-organizing feature maps. In Proceedings of 1990 ICASSP, pages 2261{2264,

Albuquerque, NM, April 1990.

28

[11] D. L. Neuho� and N. Moayeri. Tree searched vector quantization with interblock noise-

less coding. In Proceedings of 1988 Conference on Information Sciences and Systems,

pages 781{783, March 1988.

[12] N. Phamdo and N. Farvardin. Optimal detection of discrete Markov sources over dis-

crete memoryless channels - applications to combined source-channel coding. IEEE

Transactions on Information Theory, 40(1):186{193, January 1994.

[13] N. Phamdo, N. Farvardin, and T. Moriya. A uni�ed approach to tree-structured and

multi-stage vector quantization for noisy channels. IEEE Transactions on Information

Theory, 39(3):835{850, May 1993.

[14] G. Poggi. Address-predictive vector quantization of images by topology-preserving code-

book ordering. In European Transactions on Telecommunications, volume 4, pages 423{

434, July-August 1993.

[15] E. A. Riskin, L. E. Atlas, and Shyh-Rong Lay. Ordered neural maps and their ap-

plication to data compression. In Proceedings of the 1991 IEEE Workshop on Neural

Networks for Signal Processing, pages 543 { 551, 1991.

[16] E. A. Riskin, R. Ladner, R.-Y. Wang, and L. E. Atlas. Index assignment for progressive

transmission of full search vector quantization. IEEE Transactions on Image Processing,

3(3):307{312, May 1994.

[17] K. Sayood and J. Borkenhagen. Use of residual redundancy in the design of joint

source/channel coders. IEEE Transactions on Communications, 39(6):838{846, June

1991.

[18] K-H Tzou. Progressive image transmission: a review and comparison of techniques.

Optical Engineering, 26(7):581{589, July 1987.

29

[19] R.-Y. Wang. Organization of Fixed Rate Vector Quantization Codebooks. PhD thesis,

University of Washington, Seattle, WA, July 1994.

[20] R.-Y. Wang, E. A. Riskin, and R. Ladner. Codebook organization to enhance MAP

detection of weighted universal vector quantized image transmission over noisy channels.

In Proceedings Data Compression Conference, page 463, March 1994.

[21] X. Wu and K. Zhang. A better tree-structured vector quantizer. In Proceedings Data

Compression Conference, pages 392{401, Snowbird, Utah, April 1991.

[22] K. Zeger and A. Gersho. Pseudo-gray coding. IEEE Transactions on Communications,

38(12):2147{2158, December 1990.

30