adaptive distributed source coding a...

ADAPTIVE DISTRIBUTED SOURCE CODING

A DISSERTATION

SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING

AND

THE COMMITTEE ON GRADUATE STUDIES

OF STANFORD UNIVERSITY

IN PARTIAL FULFILLMENT OF THE REQUIREMENTS

FOR THE DEGREE OF

DOCTOR OF PHILOSOPHY

David P. Varodayan

March 2010

c© Copyright by David P. Varodayan 2010

All Rights Reserved

ii

Abstract

Distributed source coding, the separate encoding and joint decoding of statistically

dependent sources, has many potential applications ranging from lower complexity

capsule endoscopy to higher throughput satellite imaging. This dissertation improves

distributed source coding algorithms and the analysis of their coding performance to

handle uncertainty in the statistical dependence among sources.

We construct sequences of rate-adaptive low-density parity-check (LDPC) codes

that enable encoders to switch flexibly among coding rates in order to adapt to

arbitrary degrees of statistical dependence. These code sequences operate close to

the Slepian-Wolf bound at all rates. Rate-adaptive LDPC codes with well-designed

source degree distributions outperform commonly used rate-adaptive turbo codes.

We then consider distributed source coding in the presence of hidden variables

that parameterize the statistical dependence among sources. We derive performance

bounds for binary and multilevel models of this problem and devise coding algorithms

for both cases. Each encoder sends some portion of its source to the decoder uncoded

as doping bits. The decoder uses the sum-product algorithm to simultaneously recover

the source and the hidden statistical dependence variables. This system performs close

to the derived bounds when an appropriate doping rate is selected.

We concurrently develop techniques based on density evolution to analyze our

coding algorithms. Experiments show that our models closely approximate empirical

coding performance. This property allows us to efficiently optimize parameters of the

algorithms, such as source degree distributions and doping rates.

We finally demonstrate the application of these adaptive distributed source coding

techniques to reduced-reference video quality monitoring, multiview coding and low-

complexity video encoding.

iv

Acknowledgments

There are far too many to thank individually, but the efforts of the following were

crucial. Foremost is my advisor Prof. Bernd Girod, from whom I have learned the

arts of making research compelling and engaging. Every facet of this work reflects

his wisdom. Prof. Andrea Montanari and Prof. John Gill offered, not only thoughtful

comments on this dissertation, but also their insights into the analysis of codes.

All my colleagues in the Image, Video and Multimedia Systems group were sources

of inspiration and friendship. It was sheer enjoyment to learn from Anne Aaron,

Shantanu Rane and David Rebollo-Monedero, pioneers of distributed source coding.

Another mentor Prof. Markus Flierl showed me the ways of the academician. The

quartet Aditya Mavlankar, Yao-Chung Lin, David Chen and Keiichi Chono were my

key collaborators. We shared, not just code and ideas, but the daily ups and downs

of research life. Kelly Yilmaz made everything administrative as easy as possible.

Since arriving at Stanford, I have had the companionship and encouragement of

friends, old and new. There were Bay Area Night Game adventures with Prasad, Dan

and Justin, my longtime friends from Australia. News from Toronto was shared with

Adam, Alan, Hattie, Kris, Ming, Lin, Simon, Melissa, Andrew, Nat, Jimmy and Sam,

and in a sporadic but ongoing conversation with Jenny. I befriended many wonderful

people at Stanford, among them Erick, Joyce, Laura, Ocie, Girish, Rebecca, Yuan,

Christine and Sean, and everyone in my GSBGEN 374 T-group.

My deepest gratitude is for my family. My parents Paul and Selvi raised me

across continents to think across boundaries. My sister Florence and I share a lifelong

dialogue that influences me as a brother, son and father. Grace, my wife, brings love

and joy to my life in measures previously unimagined. I thank her for all her efforts

in support of my success. Little Oscar, appearing recently, reminds me how we learn.

v

Contents

Abstract iv

Acknowledgments v

1 Introduction 1

2 Distributed Source Coding Background 4

2.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 Lossless Coding . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.2 Lossy Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2 Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.1 Slepian-Wolf Coding . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.2 Wyner-Ziv Coding . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3 Selected Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.1 Low-Complexity Video Encoding . . . . . . . . . . . . . . . . 9

2.3.2 Multiview Coding . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3.3 Reduced-Reference Video Quality Monitoring . . . . . . . . . 12

2.4 Advanced Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.4.1 Rate Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.4.2 Side Information Adaptation . . . . . . . . . . . . . . . . . . . 14

2.4.3 Multilevel Distributed Source Coding . . . . . . . . . . . . . . 14

2.4.4 Performance Analysis and Code Design . . . . . . . . . . . . . 15

2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

vi

3 Rate Adaptation 16

3.1 Rationale for Rate-Adaptive LDPC Codes . . . . . . . . . . . . . . . 17

3.2 Construction of Rate-Adaptive LDPC Code Sequences . . . . . . . . 18

3.2.1 Rate Adaptation by Syndrome Deletion . . . . . . . . . . . . 18

3.2.2 Rate Adaptation by Syndrome Merging . . . . . . . . . . . . . 20

3.2.3 Rate-Adaptive Sequences by Syndrome Splitting . . . . . . . . 22

3.3 Analysis of Rate Adaptation . . . . . . . . . . . . . . . . . . . . . . . 23

3.4 Rate-Adaptive LDPC Coding Experimental Results . . . . . . . . . . 25

3.4.1 Performance Comparison with Other Codes . . . . . . . . . . 26

3.4.2 Performance of Rate Adaptation Analysis . . . . . . . . . . . 27

3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4 Side Information Adaptation 30

4.1 Distributed Random Dot Stereogram Coding . . . . . . . . . . . . . . 31

4.2 Model for Source and Side Information . . . . . . . . . . . . . . . . . 35

4.2.1 Definition of the Block-Candidate Model . . . . . . . . . . . . 35

4.2.2 Slepian-Wolf Bound . . . . . . . . . . . . . . . . . . . . . . . . 37

4.3 Side-Information-Adaptive Codec . . . . . . . . . . . . . . . . . . . . 38

4.3.1 Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

4.3.2 Decoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.4 Analysis of Side Information Adaptation . . . . . . . . . . . . . . . . 44

4.4.1 Factor Graph Transformation . . . . . . . . . . . . . . . . . . 44

4.4.2 Derivation of Degree Distributions . . . . . . . . . . . . . . . 45

4.4.3 Monte Carlo Simulation of Density Evolution . . . . . . . . . 48

4.5 Side-Information-Adaptive Coding Experimental Results . . . . . . . 51

4.5.1 Performance with and without Doping . . . . . . . . . . . . . 52

4.5.2 Performance under Different Block-Candidate Model Settings 53

4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5 Multilevel Side Information Adaptation 57

5.1 Challenges of Extension to Multilevel Coding . . . . . . . . . . . . . 58

5.2 Model for Multilevel Source and Side Information . . . . . . . . . . . 61

5.2.1 Multilevel Extension of the Block-Candidate Model . . . . . . 61

vii

5.2.2 Slepian-Wolf Bound . . . . . . . . . . . . . . . . . . . . . . . . 61

5.3 Multilevel Side-Information-Adaptive Codec . . . . . . . . . . . . . . 62

5.3.1 Whole Symbol Encoder . . . . . . . . . . . . . . . . . . . . . . 62

5.3.2 Whole Symbol Decoder . . . . . . . . . . . . . . . . . . . . . . 62

5.4 Analysis of Multilevel Side Information Adaptation . . . . . . . . . . 67

5.4.1 Factor Graph Transformation . . . . . . . . . . . . . . . . . . 67

5.4.2 Derivation of Degree Distributions . . . . . . . . . . . . . . . 69

5.4.3 Monte Carlo Simulation of Density Evolution . . . . . . . . . 70

5.5 Multilevel Coding Experimental Results . . . . . . . . . . . . . . . . 73

5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6 Applications 77

6.1 Reduced-Reference Video Quality Monitoring . . . . . . . . . . . . . 77

6.1.1 ITU-T J.240 Standard . . . . . . . . . . . . . . . . . . . . . . 79

6.1.2 Maximum Likelihood PSNR Estimation . . . . . . . . . . . . 80

6.1.3 Distributed Source Coding of J.240 Coefficients . . . . . . . . 81

6.2 Multiview Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

6.3 Low-Complexity Video Encoding . . . . . . . . . . . . . . . . . . . . 91

6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

7 Conclusions 99

Bibliography 102

viii

List of Tables

6.1 Codec settings for distributed source coding of J.240 coefficients . . . 83

6.2 Camera array geometry for multiview data sets Xmas and Dog . . . . 88

6.3 Average PSNR and rate for multiview coding traces . . . . . . . . . . 89

6.4 Average luminance PSNR and rate for distributed video coding traces 95

ix

List of Figures

1.1 Stereographic solar views captured by NASA’s STEREO satellites . . 1

1.2 Locations of STEREO satellites on July 13, 2007 . . . . . . . . . . . 2

2.1 Block diagrams of lossless distributed source coding . . . . . . . . . . 5

2.2 Slepian-Wolf rate region for lossless distributed source coding . . . . . 6

3.1 Rate-adaptive distributed source coding with feedback . . . . . . . . 17

3.2 Fixed rate LDPC code encoding and decoding graphs . . . . . . . . . 18

3.3 Factor graph produced by syndrome deletion . . . . . . . . . . . . . . 19

3.4 Creation of 4-cycles by adding a single edge to a factor graph . . . . . 20

3.5 Factor graph produced by cumulative syndrome deletion . . . . . . . 21

3.6 Rate-adaptive LDPC code performance . . . . . . . . . . . . . . . . . 27

3.7 Distributions of rate for rate-adaptive LDPC codes . . . . . . . . . . 28

3.8 Rate-adaptive LDPC code performance modeled by density evolution 29

4.1 Stereogram consisting of a pair of random dot images . . . . . . . . . 31

4.2 Superpositions of two random dot images . . . . . . . . . . . . . . . . 32

4.3 Asymmetric distributed source coding of random dot images . . . . . 33

4.4 Block-candidate model of statistical dependence . . . . . . . . . . . . 36

4.5 Slepian-Wolf bounds for block-candidate model . . . . . . . . . . . . 39

4.6 Factor graph for side-information-adaptive decoder . . . . . . . . . . 41

4.7 Factor graph node combinations . . . . . . . . . . . . . . . . . . . . . 42

4.8 Transformed factor graph equivalent in terms of convergence . . . . . 46

4.9 Density evolution for side-information-adaptive decoder . . . . . . . . 49

4.10 Side-information-adaptive codec performance with and without doping 52

x

4.11 Side-information-adaptive codec performance with different block sizes 54

4.12 Side-information-adaptive codec performance with different numbers

of candidates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.1 Comparison of whole symbol and bit-plane-by-bit-plane statistics . . 59

5.2 Isomorphism of binary and Gray orderings of 2-bit symbols . . . . . . 60

5.3 Nonisomorphism of binary and Gray orderings of 3-bit symbols . . . . 60

5.4 Slepian-Wolf bounds for multilevel block-candidate model . . . . . . . 63

5.5 Factor graph for multilevel decoder . . . . . . . . . . . . . . . . . . . 64

5.6 Multilevel factor graph nodes . . . . . . . . . . . . . . . . . . . . . . 65

5.7 Transformed multilevel factor graph equivalent in terms of convergence 68

5.8 Density evolution for multilevel decoder . . . . . . . . . . . . . . . . . 70

5.9 Multilevel codec performance with b = 8, c = 2 and m = 2 . . . . . . 75



5.12 Multilevel codec performance with b = 64, c = 16 and m = 2 . . . . . 76

6.1 Video transmission with quality monitoring and channel tracking . . 78

6.2 Dimension reduction projection of ITU-T J.240 standard . . . . . . . 79

6.3 Frame 0 of Foreman sequence, encoded and transcoded . . . . . . . . 81

6.4 PSNR estimation error using the J.240 standard . . . . . . . . . . . . 82

6.5 Analysis of distributed source coding of J.240 coefficients . . . . . . . 84

6.6 Coding rate for distributed source coding of J.240 coefficients . . . . . 85

6.7 Mean absolute PSNR estimation error versus estimation bit rate . . . 86

6.8 Lossy transform-domain adaptive distributed source codec . . . . . . 87

6.9 Comparison of rate-distortion performance for multiview coding . . . 89

6.10 PSNR and rate traces for multiview coding . . . . . . . . . . . . . . . 90

6.11 Stereographic reconstructions of pairs of multiview images . . . . . . 92

6.12 Comparison of rate-distortion performance for distributed video coding 94

6.13 Luminance PSNR traces for distributed video coding . . . . . . . . . 96

6.14 Rate traces for distributed video coding . . . . . . . . . . . . . . . . . 97

6.15 Reconstructions of distributed video coding frames . . . . . . . . . . 98

xi

Chapter 1

Introduction

Consider the source coding (or data compression) of the images of the sun in Fig. 1.1.

These stereographic views were captured by a pair of satellites, part of NASA’s Solar

Terrestrial Relations Observatory (STEREO), on July 13, 2007 [92]. The locations of

the satellites, named STEREO Behind and STEREO Ahead, on that day are plotted

in Fig. 1.2. At roughly 50 million km apart, the satellites did not communicate with

each other and maintained limited communication with Earth. Therefore, the source

coding of the images for their reconstruction on Earth could involve separate encoders

and a joint decoder; a setting known as distributed source coding.

Figure 1.1: Stereographic solar views captured by NASA’s STEREO satellites [131].

1

CHAPTER 1. INTRODUCTION 2

Figure 1.2: Locations of STEREO Behind (B) and STEREO Ahead (A) satellites on July13, 2007. The coordinates are Heliocentric Earth Ecliptic (HEE) and Geocentric SolarEcliptic (GSE) with axes labeled in astronomical units [131].

Distributed source coding is proposed for many other practical applications [135].

An example is capsule endoscopy, the diagnostic imaging of the small intestine using

a pill-sized wireless camera swallowed by the patient. Due to power and memory

constraints, the capsule cannot encode consecutive frames of video jointly. The re-

ceiver, on the other hand, can decode the frames jointly because it is unconstrained

in complexity, by virtue of being located outside the patient’s body.

This dissertation studies the distributed coding of statistically dependent sources

under uncertainty about their statistical dependence. Our contributions are encom-

passed by the title adaptive distributed source coding.

• Rate adaptation is a technique for an encoder to adapt to uncertainty in the

degree of statistical dependence. Our construction of rate-adaptive low-density

parity-check (LDPC) codes allows an encoder to switch flexibly between en-

coding bit rates. If there is feedback from decoder to encoder, a rate-adaptive

encoder need not know the degree of statistical dependence in advance.

CHAPTER 1. INTRODUCTION 3

• Side information adaptation is a technique for the coding system to adapt to

hidden variables that parameterize the statistical dependence. We formulate

models for both binary and multilevel source signals with hidden dependence

and compute performance bounds for them. We develop encoders that send some

portion of their sources uncoded as doping bits and decoders that discover the

hidden variables (while decoding the source) using the sum-product algorithm.

• The analysis of rate and side information adaptation enables the designer to

adapt the system to specific statistics or to make it robust against uncertain

statistics. We provide density evolution techniques to estimate and optimize

performance based on the statistical model and coding algorithm parameters,

such as LDPC code degree distributions and doping rates.

The dissertation makes the first and second contributions in turn, while devel-

oping the third contribution concurrently. Chapter 2 reviews the theory, practice

and applications of distributed source coding, with a focus on precursors to our work.

Chapter 3 motivates the need for rate-adaptive LDPC codes, describes their construc-

tion and obtains the density evolution method for their analysis. We establish that

these codes operate near established information-theoretic performance bounds and

that density evolution models the performance of different codes accurately. Chap-

ter 4 introduces side information adaptation using the simple example of random

dot stereograms. We formalize a binary model for hidden statistical dependence and

derive theoretical performance bounds. Coding algorithms and analysis methods are

presented and their performance is evaluated with respect to each other and the

bounds for various settings of the model. Chapter 5 extends side information adap-

tation to multilevel signals, starting by addressing the additional challenges. As in

the preceding chapter, we continue our discussion with a model, performance bounds,

coding algorithms, analysis methods and experimental validation. Chapter 6 applies

the techniques developed in this dissertation to problems in reduced-reference video

quality monitoring, multiview coding and low-complexity video encoding. We show-

case a different aspect of performance for each application. Chapter 7 concludes the

dissertation and offers avenues for further research.

Chapter 2

Distributed Source Coding

Background

Distributed source coding is the data compression of two (or more) sources, or one (or

more) sources with additional side information available at decoder only, such that

the encoders for each source are separate. A single joint decoder receives all the en-

codings, exploits statistical dependencies, and reconstructs the sources, if applicable,

in conjunction with the side information. Fig. 2.1 shows block diagrams for lossless

distributed source coding with two signals. The configuration in Fig. 2.1(b), called

source coding with side information at the decoder, is a special case (often referred

to as the asymmetric case) of the configuration in Fig. 2.1(a).

This chapter presents a survey of distributed source coding. Section 2.1 and Sec-

tion 2.2 outline theoretical foundations and elementary practical techniques, respec-

tively, for both lossless and lossy coding. The state-of-the-art for selected applications

is described in Section 2.3. In Section 2.4, we address recent advances in distributed

source coding techniques, upon which this dissertation makes novel and unifying con-

tributions. Our treatment in this chapter draws, in part, from both [50] and [75].

4

CHAPTER 2. DISTRIBUTED SOURCE CODING BACKGROUND 5

JointDecoder

SeparateEncodersource X reconstruction X

SeparateEncodersource Y reconstruction Y

RX

RY

(a)

Encoder Decodersource X

side information Y

reconstruction XRX

(b)

Figure 2.1: Block diagrams of lossless distributed source coding with two signals: (a) sepa-rate encoding and joint decoding and (b) source coding with side information at the decoder,also known as the asymmetric case of (a).

2.1 Theory

2.1.1 Lossless Coding

Consider the configuration in Fig. 2.1(a), in which X and Y are each finite-alphabet

random sequences of independent and identically distributed samples. Separate en-

coding and separate decoding enable compression of X and Y without loss to rates

RX ≥ H(X) and RY ≥ H(Y ), respectively [175]. Under separate encoding and

joint decoding, though, the Slepian-Wolf theorem [177] shows that the rate region

(tolerating an arbitrarily small error probability) expands to

RX ≥ H(X|Y ), RY ≥ H(Y |X), RX +RY ≥ H(X, Y ), (2.1)

as shown in Fig. 2.2. Remarkably, the sum rate boundary RX +RY = H(X, Y ) is just

as good as the achievable rate for centralized coding (i.e., joint encoding and joint

decoding) of X and Y . At the corner points A and B in Fig. 2.2 the problem reduces

to the asymmetric case of source coding with side information at the decoder, as


H(X|Y) H(X)

H(Y)

H(Y|X)

RX

RY

RX +RY =H(X,Y)

A

B

Separate EncodingSeparate Decoding

Separate EncodingJoint Decoding

Figure 2.2: Slepian-Wolf rate region for lossless distributed source coding of two sources.

depicted in Fig. 2.1(b). This is because at point A, for example, the rate RY = H(Y )

makes Y available to the decoder via conventional lossless coding.

Several variants of lossless distributed source coding admit information-theoretic

bounds. In [12] and [94], only the recovery of X and X+Y modulo 2, respectively, is

important. In [224], the decoder still recovers X and Y , but there is an additional sep-

arate encoder which sees yet another statistically dependent source. Another branch

of study, zero-error distributed source coding, requires that the error probability be

not just vanishing, but strictly zero [220, 96, 95, 176, 233, 200].

2.1.2 Lossy Coding

A straightforward achievable rate-distortion region for lossy distributed source coding

of two sources, stated in [27, 201, 26] and independently in [87], results from combining

quantization with the achievable rate region for lossless coding. The extension to more

than two sources is in [83]. For memoryless Gaussian sources and mean squared error

distortion, an outer bound [212, 211] shows that this region is tight [213]. In general,

there are only loose outer bounds [72, 73].


In the special case of lossy source coding with side information at the decoder,

the Wyner-Ziv theorem provides a single-letter characterization of the rate-distortion

function [226, 225]. In particular, this theorem (and, several years later, the result

in [213]) show that, in the Gaussian source and mean squared error distortion case,

there is no rate loss incurred by the encoder not knowing the side information. 1

The rate loss, for general memoryless sources and mean squared error distortion, is

bounded below 12

bit/sample [239].

Extensions of the Wyner-Ziv problem consider successive refinement [182, 183] and

the possible absence of side information [84]. Another extension called noisy Wyner-

Ziv coding concerns encoding a noisy observation of the source, and decoding with

the help of side information [232, 61, 62, 51, 54]. Some noisy Wyner-Ziv problems

revert to noiseless cases when modified distortion measures are used [221, 114, 115].

2.2 Techniques

2.2.1 Slepian-Wolf Coding

Although the proof of the Slepian-Wolf theorem is nonconstructive, the method of

proof by random binning (see also [45]) provides a connection to channel coding.

Consider again two statistically dependent binary sources X and Y related through

a hypothetical error channel. A linear channel code that achieves capacity for this

channel also achieves the Slepian-Wolf bound [223]. In the asymmetric case (i.e.,

the source coding of X with side information Y at the decoder), the Slepian-Wolf

encoder generates the syndrome of X with respect to the code. This encoding is

a binning operation since the entire coset of X maps to the same syndrome. The

decoder recovers X by selecting the coset member that is jointly typical with Y . A

code used this way for distributed source coding is said to be in syndrome-generating

form. An alternative usage of a channel code is called the parity-generating form, for

which there is no guarantee that the distributed source coding performance matches

the channel coding performance. In the asymmetric case, this approach encodes X

into parity bits with respect to a systematic channel code. The decoder uses the

parity bits to correct the hypothetical errors in Y , and so recovers X.

1The dirty-paper theorem for channel coding [44] is the dual of this result [184, 24, 137].


The earliest application of channel coding techniques to source coding predates the

Slepian-Wolf formulation [29] (cited in [85].) Despite such precursors, it was decades

before the revival of the technique for distributed source coding in the asymmet-

ric [139, 144, 215] and symmetric cases [140, 142, 141, 138] using convolutional/trellis

codes (see [216].) Modern channel codes like turbo codes [28] and low-density parity-

check (LDPC) codes [64] provide substantially better performance. Turbo codes are

applied mostly in parity-generating form for both asymmetric [22, 128, 3] and symmet-

ric coding [65, 68]. Although turbo codes in syndrome-generating form are possible,

decoding the syndrome requires new trellis constructions [191, 190, 163]. In contrast,

LDPC codes allow straightforward syndrome decoding for asymmetric [117, 118, 100]

and symmetric coding [168, 169, 170, 43, 74]. Nevertheless, systematic LDPC codes

are also applied in parity-generating form [166, 167]. Symmetric usage of LDPC codes

is extended to multiple sources in [180, 181, 167].

We defer discussion of several additional capabilities, namely, rate adaptation,

side information adaptation, multilevel distributed source coding, and performance

analysis and code design, to Section 2.4, by which point we already motivate their

use in applications in Section 2.3. Among advanced topics that we do not describe

in detail are distributed zero-error coding [13, 242], distributed (quasi-)arithmetic

coding [17, 77, 125] and joint source-channel coding using distributed source coding

techniques [66, 249, 129, 69, 119].

2.2.2 Wyner-Ziv Coding

A simple construction of a Wyner-Ziv encoder is the concatenation of a vector quan-

tizer (possibly having noncontiguous cells) and a Slepian-Wolf encoder. The proof of

the converse in [226] suggests that this separation incurs no loss in performance with

respect to the Wyner-Ziv bound in the limit of large block length.

Structured quantizers with noncontiguous cells are implemented using nested lat-

tices [139, 173, 99, 228, 116] (based on information-theoretic results for jointly Gaus-

sian source and side information [240, 241]) or trellis-coded quantization [235, 227].

Another approach generalizes the Lloyd algorithm [120] to find locally optimal vector

quantizers [57, 34, 159, 58]. Globally optimal contiguous-cell quantizers are designed

in [130], but are shown to be nonoptimal in general [56]. Pleasingly, for quantization


at high rates, lattices (with contiguous cells) are optimal [154, 157]. Quantizer design

for the noisy Wyner-Ziv problem is treated in [158, 155, 156, 153].

2.3 Selected Applications

The distributed source coding applications reviewed in this section motivate the tech-

nical contributions of this dissertation and are revisited in Chapter 6. Beyond our se-

lection, there is much related work in hyperspectral image coding [103, 23, 192, 40, 41],

array audio coding [124, 49, 165], error-resilient video coding [171, 172, 229], sys-

tematic lossy error protection of video [143, 6, 151, 214, 150, 152], biometric secu-

rity [126, 52, 53, 186, 185], media authentication [108, 207, 105, 110, 111, 112] and

media tampering localization [107, 109, 145, 106, 189, 203].

2.3.1 Low-Complexity Video Encoding

International standards for digital video coding (including ITU-T H.264/MPEG-4

AVC [217]) prescribe syntax for motion-compensated predictive coding. The en-

coder exploits the spatiotemporal dependencies of the video signal to produce a bit

stream as free of redundancy as possible. A small proportion of frames (called key

frames) are coded separately, while the remainder (called predictive frames) consist

of blocks that are either skipped, coded separately or coded as residuals with re-

spect to prediction blocks. A prediction block is selected by motion search within

one or more previously reconstructed frames. In implementations of predictive video

coding, the encoder is complex (with the bulk of the computation due to motion

search) and typically consumes five to ten times the resources needed by the decoder.

This unequal distribution of complexity is appropriate for traditional applications of

broadcast and storage in which video is encoded once and may be decoded thousands

or millions of times. But modern encoding devices, like capsule endoscopes, camera

phones and wireless surveillance cameras, are often severely resource-constrained and

consequently ill-suited to motion-compensated predictive coding.

Distributed source coding offers an alternative approach, in which the responsibil-

ity for exploiting temporal redundancy is transferred from encoder to decoder. The

key frames are still coded separately, but the remaining frames (called Wyner-Ziv


frames) are encoded separately and decoded using side information constructed from

previously reconstructed frames. In this way, the encoder avoids the complexity of

motion search between frames. The decoder, in contrast, takes on additional burden

in constructing statistically dependent side information (perhaps by motion search)

and decoding the Wyner-Ziv encoding. Amazingly, this idea described first in [222]

precedes the standardization of predictive video coding. Its rediscovery independently

in [11] and [147] is responsible for the renaissance in research into distributed source

coding. In the rest of this section, we outline the main ideas of three distributed video

codecs.

In the PRISM codec [147, 149, 148, 146], each block of a Wyner-Ziv frame is

either skipped, coded separately or Wyner-Ziv coded, depending on its correlation

with the colocated block in the previous reconstructed frame. Encoding a Wyner-Ziv

block produces a syndrome with respect to a Bose-Chaudhuri-Hocquenghem (BCH)

code [86, 30] and a cyclic redundancy check (CRC) checksum. For each Wyner-

Ziv block, the decoder constructs multiple candidates of side information from a

search range within the previous reconstructed frame. The syndrome is decoded with

reference to the side information candidates one-by-one until the CRC checksum of the

encoded and recovered blocks match. Block-by-block Wyner-Ziv coding necessitates

short length codes, which are far from optimal, and a significant amount of auxiliary

CRC data. To compensate for poor coding performance, some versions of PRISM

revert to a limited motion search at the encoder [146].

The Wyner-Ziv video codec [11, 8, 9, 7] is so called because it adheres more closely

to distributed source coding principles. All blocks of a Wyner-Ziv frame are encoded

together, without reference to other frames, as parity bits of long rate-compatible

turbo codes [164]. Such codes enable the encoder to increment the coding rate until

the decoder signals sufficiency via a feedback channel. The side information for a

Wyner-Ziv frame is constructed at the decoder by motion-compensated interpolation

of previously reconstructed key frames. The transmission of either auxiliary hashes [5]

or uncoded less significant bit planes [4] of the Wyner-Ziv blocks assists motion-

compensated extrapolation at the decoder, and therefore reduces the need for frequent

key frames. In one version of the codec, Wyner-Ziv encoding of the residual of a frame

with respect to the previous frame relaxes strict separate encoding [10]. This paper


also compares rate-compatible turbo codes with rate-adaptive LDPC codes [204, 205],

from Chapter 3 of this dissertation, and finds the rate-adaptive LDPC codes superior.

The DISCOVER codec follows the architecture of the Wyner-Ziv video codec but

refines its components to improve performance and practicality [16]. Particular ad-

vances are made in the construction of side information [18, 19, 132, 21], the modeling

of the dependence between source and side information [31, 32, 33] and the recon-

struction of the source [98]. It bears mentioning that the best DISCOVER video

coding performance is achieved using the rate-adaptive LDPC codes of this disserta-

tion, rather than rate-compatible turbo codes or a subsequent variant of rate-adaptive

LDPC codes [20].

Distributed video coding has yet to match the coding performance of motion-

compensated predictive coding. Towards this goal, the principal investigators of the

DISCOVER project identify “finding the best side information (or predictor) at the

decoder” as a key task [78]. Chapter 4 of this dissertation contributes the idea of

binary side information adaptation and Chapter 5 extends it to the multilevel case.

Using these techniques, the decoder of a distributed video codec learns the best

motion-compensated side information among a vast number candidates [206].

2.3.2 Multiview Coding

Multiview images and video, captured by a network of cameras, arise in applications

like light field imaging [219, 218] and free viewpoint television [63, 127, 193, 194].

When the camera network is limited in communication, centralized source coding

(addressed in the multiview video coding extension [39] to H.264/AVC) is ineffective

because the uncoded views are transmitted to a joint encoder, creating a bottleneck

as the number of views grows. Under distributed source coding, in contrast, the views

are first coded by separate encoders attached to the cameras and then transmitted

to a joint decoder.

Distributed multiview image coding exploits view dependence in much the same

way as distributed single-view video coding does temporal dependence. For multiview

image coding, the main differences are that the image model is geometric (based on

the camera arrangement) and that the encoding is strictly separate. The Wyner-Ziv

video and PRISM codecs are modified accordingly in [250] and [198], respectively.


Distributed multiview video coding exploits both temporal and view dependence,

the former per view at each of the separate encoders and the latter at the joint decoder.

Unsurprisingly, there is a trade-off between the maximum possible temporal and

view coding gains [60]. Out of the potential view coding gain, the amount achieved

depends on the quality of the interpolated or extrapolated side information views at

the decoder. Side information for a certain time instant is constructed from other

views at the same instant based on disparity [59], affine [79] and homography [133, 15]

models. Camera geometry provides an additional epipolar constraint that reduces the

parameter space of these models [178, 179, 238]. The synthesis of a side information

view from a time duration of views produces even better coding results [237, 187, 55].

For two views, the distributed multiview video codec in [236] is the first to outperform

separate encoding and separate decoding using H.264/AVC.

Just as for low-complexity video encoding, this dissertation offers powerful adap-

tive distributed source coding tools for multiview coding [209, 210, 208, 35, 37, 36].

2.3.3 Reduced-Reference Video Quality Monitoring

In digital video broadcast systems, a content server typically transmits video to client

devices over imperfect links or insecure networks, while subject to stringent delay and

latency requirements. The first step towards ensuring quality of service is monitoring

the fidelity of the received video without significantly impacting the primary trans-

mission. No-reference methods estimate the video quality at each client based on the

received video alone [230]. Reduced-reference methods make use of a supplementary

low-rate bit stream for more accurate quality monitoring [231]. The ITU-T J.240

standard specifies this bit stream as a projection of the video signal after whitening

in both space and Walsh-Hadamard transform domains [2, 93]. A comparison of the

J.240 projections of the transmitted and received video yields an estimate of the peak

signal-to-noise ratio (PSNR).

Conventional source coding of the J.240 projection achieves only limited gains

because its whitened coefficients have large variance. In contrast, distributed source

coding of the transmitted video’s J.240 projection using the received video as side

information at the decoder is effective [42]. Similar work on distributed source coding

of random projections and projections based on perceptual metrics is reported in [202]


and [188], respectively. Once a client device deems its received video quality to be

unacceptable, it can request retransmission of packets from the server [104]. If the

client device is too resource-constrained to perform the decoding and analysis itself,

it can instead feed back to the server its distributed-source-coded J.240 projection for

decoding with side information consisting of the transmitted (or even the original)

video [113].

The rate-adaptive LDPC codes of Chapter 3 feature in most of these systems [42,

202, 113, 188]. In this dissertation, we further describe how side information adap-

tation of Chapter 4 and its multilevel extension of Chapter 5 add the capability

of channel tracking to reduced-reference video quality monitoring with distributed

source codes.

2.4 Advanced Techniques

The applications described in the previous section demand more from distributed

source codes than just good coding performance in the Slepian-Wolf and Wyner-Ziv

settings. We now motivate the contributions of this dissertation and place them in

the context of precursors and (if applicable) subsequent work.

2.4.1 Rate Adaptation

In practice, the degree of statistical dependence between source and side informa-

tion is varying in time and unknown in advance. This is the case, for example, for

J.240 projections of transmitted and received video. So, it is better to model the

bound on coding performance of asymmetric distributed source coding as the varying

conditional entropy rate H(X|Y) of source and side information sequences X and

Y, respectively, rather than the fixed conditional entropy H(X|Y ) of their samples.

Codes that can adapt their coding rates incrementally in response to varying statis-

tics are more practical. In parity-generating form, convolutional and turbo codes are

punctured to different rates with good performance in [81] and [164, 82], respectively.

But removing syndrome bits of LDPC codes in syndrome-generating form leads to

performance degradation at lower rates [80, 136, 174, 101, 190]. The rate-adaptive


sequences of syndrome-generating codes developed in [38] are based on product ac-

cumulate and extended Hamming accumulate codes [102, 89]. We construct the first

rate-adaptive LDPC codes in [204, 205]. Rate adaptation for convolutional and turbo

codes in syndrome-generating form is provided for the asymmetric case in [163] and

the symmetric case in [199]. A different approach towards rate adaptation for LDPC

codes uses a combination of syndrome and parity bits [90].

2.4.2 Side Information Adaptation

Realistic models of the statistical dependence between source and side information

often involve hidden variables. For example, the motion vectors that relate consecu-

tive frames of video are unknown a priori at the decoder of a distributed video codec.

Side information adaptation encompasses methods for learning the hidden variables

and adapting the side information accordingly, and doing so more efficiently than by

exhaustive search. In [70, 245, 71, 67, 247], the statistical dependence is through a

hidden Markov model, so side information adaptation with respect to these variables

is by the Baum-Welch algorithm [25]. We make use of the expectation maximiza-

tion (EM) generalization [47] to decode images related by hidden disparity [209, 208]

and motion [205], and (in work not reported in this dissertation) through contrast

and brightness adjustment [105, 106], cropping and resizing [110], and affine warp-

ing [111, 112].

2.4.3 Multilevel Distributed Source Coding

Much of distributed source coding research, as in channel coding, concerns binary

data, even though signals are frequently quantized to symbols of more than 2 levels.

One approach to multilevel distributed source coding uses codes on finite fields of

order greater than 2 [215, 3], but such codes are cumbersome to design for different

numbers of levels. Another method for coding a multilevel source is applying binary

distributed source coding bit-plane-by-bit-plane, using each decoded bit plane as ad-

ditional side information for the decoding of subsequent bit planes [7, 38]. Although

this technique allows side information adaptation per bit plane, statistical dependence

through hidden variables usually spans all bit planes. A third way is to encode all


the bit planes with a single binary code, but to decode them as symbols using soft

bit-to-symbol and symbol-to-bit mapping [243, 244, 246, 248]. We take this approach

in extending side information adaptation from binary to multilevel [210].

2.4.4 Performance Analysis and Code Design

The structure of LDPC codes, both global [123] and local [48], affects how closely

their performance approaches information-theoretic bounds. Assessing the codes em-

pirically slows down the design process because of the complexity of the iterative

message-passing decoding algorithm [64]. Faster methods for performance analy-

sis consider only parameters of the global structure [121, 197, 161]. Density evolu-

tion [161] summarizes an LDPC code by its degree distributions and evaluates the

convergence of message densities under message passing. This technique is used to

design high-performance LDPC codes with optimized irregular degree distributions

for both channel coding [160, 14] and distributed source coding [117, 180, 181]. In this

dissertation, we use density evolution to analyze and design, not just LDPC codes,

but also all the adaptive distributed source codes of our construction. In fact, density

evolution is applicable to all instances of the sum-product algorithm [97], known also

as loopy belief propagation [134].

2.5 Summary

This chapter reviews distributed source coding, starting with the theory and elemen-

tary techniques of both lossless and lossy coding. We then cover research progress

on three applications, namely low-complexity video encoding, multiview coding and

reduced-reference video quality monitoring. Finally, we describe recent advances in

technique, motivated by the needs of the applications, in the areas of rate adaptation,

side information adaptation, multilevel coding, and performance analysis and code

design. Our contributions in Chapters 3 to 5 of this dissertation build upon and unify

elements of this literature. In Chapter 6, we revisit the three applications reviewed

in this chapter and apply our novel techniques.

Chapter 3

Rate Adaptation

This chapter studies the construction of sequences of rate-adaptive LDPC codes for

distributed source coding.1 Each code in a rate-adaptive sequence operates at a

different rate, such that the following property holds: a code creates encoded bits

that form a subsequence of the encoded bits created by any other code of higher rate.

The encoder, using these codes, could therefore increase its coding rate on-the-fly

simply by sending an additional increment of encoded bits. In this way, the encoder

adapts its rate to handle any level of statistical dependence between the source and

the side information.

Section 3.1 motivates the pursuit of rate-adaptive LDPC codes with regard to

practical application of distributed source coding, the codes’ performance and their

amenability to analysis. In Section 3.2, we compare several approaches to construct-

ing sequences of such codes. We find that a novel construction based on syndrome

splitting offers coding performance close to the Slepian-Wolf bound over the entire

range of rates. In Section 3.3, we derive the degree distributions of these codes so that

the technique of density evolution can model their rate adaptation. The experimental

results in Section 3.4 confirm both the high performance of the rate-adaptive LDPC

codes and the accuracy of the density evolution analysis.

1In our early work [204, 205], we used the term low-density parity-check accumulate (LDPCA) codes.This was replaced by rate-adaptive LDPC codes to emphasize the function rather than the structure.

16

CHAPTER 3. RATE ADAPTATION 17

Rate-AdaptiveEncoder

Rate-AdaptiveDecodersource X

side information Y

rate control feedback

encodedincrements

reconstruction X

Figure 3.1: Rate-adaptive distributed source coding with feedback. The encoder sendsincrements of encoded data to the decoder until the decoder signals that the coding rate issufficient using the rate control feedback channel.

3.1 Rationale for Rate-Adaptive LDPC Codes

Rate-adaptive codes make distributed source coding practical for nonergodic sources,

while rate-adaptive LDPC codes, in particular, offer both very good coding perfor-

mance and a method of analysis through density evolution.

The source X and side information Y, obtained directly or through projection

from natural images or video, are almost always jointly nonergodic. The encoder’s

appropriate coding rate, which must exceed the conditional entropy rate H(X|Y),

may vary substantially and without prior notice. Codes with the rate-adaptive prop-

erty permit the encoder to flexibly increase its coding rate on-the-fly. Moreover, when

rate-adaptive codes are used in conjunction with a feedback channel from the decoder

as in Fig. 3.1, the encoder can determine the proper coding rate Radaptive as follows:

1. the encoder sends coded bits corresponding to the code of lowest rate Radaptive;

2. the decoder attempts decoding and signals via the feedback channel whether

decoding is successful;

3. if decoding is unsuccessful, the encoder increases rate Radaptive by sending an

additional increment of encoded bits;

4. Steps 2 and 3 repeat until decoding is successful.

In Step 3, we deem decoding to be successful if the decoded bits can be re-encoded

into a bit stream identical to that received from the encoder.


S1

S2

S3

S4

X1

X2

X3

X4

X5

X6

X7

X8

(a)

S1

S2

S3

S4

Y1

Y2

Y3

Y4

Y5

Y6

Y7

Y8

(b)

Figure 3.2: Fixed rate LDPC code (a) encoding bipartite and (b) decoding factor graphs.

Although turbo codes exist in rate-adaptive forms [164, 3, 82], LDPC codes offer

better fixed rate coding performance [117]. Moreover, the performance of LDPC

codes is easier to model via density evolution than that of turbo codes. For these

reasons, we develop a construction for sequences of rate-adaptive LDPC codes.

3.2 Construction of Rate-Adaptive LDPC Code Sequences

The encoder for a fixed rate LDPC code in syndrome-generating form performs bin-

ning in the following way. Using a bipartite graph like the one in Fig. 3.2(a), it encodes

the source X = (X1, X2, . . .) into syndrome bits denoted by the vector (S1, S2, . . .),

such that each syndrome bit is the modulo 2 sum of its neighbors. The decoder re-

covers the source from the syndrome bits and the side information Y = (Y1, Y2, . . .)

by running the sum-product algorithm on a factor graph like the one in Fig. 3.2(b).

In this section, we compare ways for the encoder to extend a mother LDPC code of

fixed rate into a sequence of rate-adaptive codes.

3.2.1 Rate Adaptation by Syndrome Deletion

Starting with a high rate code as a mother code, a naıve way to obtain lower rate

codes is by syndrome deletion. This technique is rate adaptive because, among any


S1

S2

S3

S4

S5

S6

S7

S8

(a)

S2

S4

S6

S8

(b)

Figure 3.3: (a) Factor graph of a regular degree 3 LDPC code of rate 1. This graph containsno 4-cycles. (b) The factor graph of the rate 1

2 code produced by syndrome deletion. Somesource nodes are unconnected or just singly connected, stifling the convergence of the sum-product algorithm.

pair of codes so created, the lower rate code’s syndrome is a subsequence of the higher

rate code’s syndrome. Unfortunately, deletion produces poor codes because it results

in the loss of edges in the graphs of the lower rate codes. The factor graphs of the

lowest rate codes ultimately contain unconnected or singly connected source nodes,

as shown in the example in Fig. 3.3. This poorly connected factor graph is unsuitable

for decoding using the sum-product algorithm.

To mitigate the loss of edges at lower rates, one might increase the number of

edges in the mother code. But more edges degrade the performance of this code

because they introduce shorter cycles. Short cycles in the factor graph allow unreliable

information to recirculate via the sum-product algorithm and become unduly believed.

For instance, the factor graph in Fig. 3.3(a) contains no 4-cycles. It can be shown by

inspection that adding any single edge to this graph creates a 4-cycle. Three examples

are shown in Fig. 3.4. Furthermore, increasing the number of edges in the mother

code increases the decoding complexity of the sum-product algorithm linearly.


(a) (b) (c)

Figure 3.4: Creation of 4-cycles by adding a single edge marked in blue to the factor graph inFig. 3.3(a). Even though the original graph contains no 4-cycles, we can show by inspectionthat any single edge added to the graph creates a 4-cycle, completed by the edges markedin red.

3.2.2 Rate Adaptation by Syndrome Merging

The key idea in achieving superior rate adaptation is to reduce the number of syn-

drome nodes, not by deletion but by merging. Syndrome merging preserves edges in

the factor graph and thereby keep the sum-product algorithm effective at lower rates.

We begin with a transformation of the encoding into an equivalent representa-

tion. Instead of sending the syndrome, the encoder sends the modulo 2 cumulative

syndrome2, denoted as the vector (C1, C2, . . .). There is a one-to-one correspondence

between the representations, since Ci = S0 + S1 + · · ·+ Si and Si = Ci +Ci−1, where

all sums are modulo 2.

The starting point of code design is once again a mother LDPC code of high rate.

But now lower rate codes are obtained by cumulative syndrome deletion, instead

of syndrome deletion. Rate adaptation is guaranteed because, among any pair of

codes so created, the lower rate code’s cumulative syndrome is a subsequence of the

higher rate code’s cumulative syndrome. In the following, we argue that deletion of

2The cumulative syndrome bits can be recast straightforwardly as the parity bits of an extended IrregularRepeat Accumulate (eIRA) channel code [234]. But in the channel coding scenario, the parity bits are subjectto a noisy channel and so the eIRA decoding graph represents them as degree 2 nodes. For this reason, weavoid conflating the concepts of rate-adaptive LDPC codes and eIRA codes.


S1=C1

S2=C2+C1

S3=C3+C2

S4=C4+C3

S5=C5+C4

S6=C6+C5

S7=C7+C6

S8=C8+C7

(a)

S1+S2=C2

S3+S4=C4+C2

S5+S6=C6+C4

S7+S8=C8+C6

(b)

Figure 3.5: (a) Factor graph of a regular degree 3 LDPC code of rate 1. (b) Factor graph ofthe rate 1

2 code produced by cumulative syndrome deletion at the encoder. Since syndromenodes are merged at the decoder, not deleted, all edges have been preserved and the sum-product algorithm remains effective.

cumulative syndrome bits at the encoder is equivalent to merging syndrome nodes at

the decoder.

Deleting the cumulative syndrome value Ci from the vector means that the syn-

drome bits Si = Ci + Ci−1 and Si+1 = Ci+1 + Ci are lost at the decoder. But a new

syndrome bit Si+Si+1 = Ci+1 +Ci−1 is still recoverable. In the factor graph, this new

bit is represented as a merged syndrome node with neighbors consisting of the union

of the neighbors of the lost nodes minus their intersection, because the new bit is the

modulo 2 sum of the pair of lost bits. In general, deleting cumulative syndrome values

Ci, Ci+1, . . . , Cj−1 merges the following syndrome value: Si+Si+1+· · ·+Sj = Cj+Ci−1.

Fig. 3.5 shows an example in which merging syndrome nodes by deleting the odd-

numbered cumulative syndrome values preserves all the edges in the factor graph.

Note that sometimes merging a pair of syndrome nodes can lead to lost edges.

This is the case when the pair shares one or more common neighbors. Although these

cases are infrequent, it is difficult to design the mother code so that no edges are

lost over several merging steps. In the next section, we propose a code construction

that avoids edge loss by generating codes from lowest rate to highest via syndrome

splitting, the inverse operation of merging.


3.2.3 Rate-Adaptive Sequences by Syndrome Splitting

We propose the following recipe to construct a good sequence of codes, for which no

edges are lost across all factor graphs:

1. select the code length n;

2. select the encoded data increment size k to be a factor of n;

3. set a code counter t = 1 and build a low rate mother LDPC code of source

coding rate Radaptive = kn

with n source nodes and k syndrome nodes;

4. build a higher rate code of rate Radaptive = (t + 1) kn

by splitting the k largest

degree syndrome nodes of the code of rate Radaptive = t kn

into pairs of nodes with

equal or consecutive degree, and increment t;

5. repeat Step 4 until the syndrome-generating matrix has full rank n.

In Step 2, we choose k to be a factor of n so that the sequence includes a code of

rate 1.

Step 3 uses the progressive edge growth method of [88, 122] to build the mother

code of desired edge-perspective source degree distribution λ(ω).3 This method cre-

ates as few short cycles as possible and also concentrates the edge-perspective syn-

drome degree distribution ρ(ω) to a single degree or a pair of consecutive degrees.4

Every splitting of k nodes in Step 4 corresponds to the transmission of an increment

of k cumulative syndrome bits. We schedule the increments to split the syndrome

nodes from largest to smallest degree into pairs of nodes with equal or consecutive

degrees, in order to retain the concentration of the syndrome degree distribution

around a limited number of values. Creating higher rate codes by splitting syndrome

nodes offers a couple of advantages versus creating lower rate codes by syndrome

merging. No edges are lost in the factor graph as long as the set of neighbors of a

split node is partitioned into the sets of neighbors of the resulting pair of nodes, and so

the source degree distribution λ(ω) is guaranteed to be invariant. Secondly, splitting

3In the edge-perspective source degree distribution polynomial λ(ω), the coefficient of ωd−1 is the fractionof source-syndrome edges connected to source nodes of degree d.

4In the edge-perspective syndrome degree distribution polynomial ρ(ω), the coefficient of ωd−1 is thefraction of source-syndrome edges connected to syndrome nodes of degree d.


a syndrome node can only remove cycles from the factor graph, not introduce them.

Therefore, this construction also guarantees that there are no additional cycles in the

sequence of factor graphs beyond the ones designed by progressive edge growth in the

original high performance mother code.

The full rank condition of Step 5 ensures that the code of highest rate can always

be decoded using Gaussian elimination regardless of the side information. The highest

coding rate is at least 1 because the syndrome nodes must number greater than or

equal to the number of source nodes, but usually the code of rate 1 suffices.

3.3 Analysis of Rate Adaptation

The method of density evolution can determine whether an LDPC code converges

given the statistical dependency between the source and side information, using just

its source and syndrome degree distributions [161]. The idea in this extension is

to test the convergence of all codes in the rate-adaptive sequence, and estimate the

coding rate according to the lowest rate code that converges. In order to reap the

full complexity savings of density evolution, the degree distributions of all the codes

must be derived without actually generating any of their graphs. The mother code’s

edge-perspective source degree distribution λ(ω) is a design choice of the progressive

edge growth method. Then, for the codes of rate Radaptive = t kn, we obtain the edge-

perspective source and syndrome degree distributions, λt(ω) and ρt(ω), respectively.

Source Degree Distribution

The construction ensures that the edge-perspective source degree distribution is in-

variant,

λt(ω) = λ(ω). (3.1)


Syndrome Degree Distribution

The syndrome degree distributions are derived inductively. The mother code’s edge-

perspective syndrome degree distribution is concentrated on at most two consecutive

values, say d and d+ 1, so we can write

ρ1(ω) = ρ(ω) = (1− α)ωd−1 + αωd, where integer d > 0 and 0 ≤ α < 1. (3.2)

and derive the unique solution

d = bdc (3.3)

α = (d− bdc)bdc+ 1

d, (3.4)

using the equality∫ 1

0ρ(ω)dω = k

n

∫ 1

0λ(ω)dω and where d =

(kn

∫ 1

0λ(ω)dω

)−1

[162].

The inductive step uses to ρt(ω) to derive ρt+1(ω), the edge-perspective syndrome

degree distributions of the codes of rate Radaptive = t kn

and Radaptive = (t + 1) kn,

respectively. In this case, it is easier to work with their node-perspective syndrome

degree distributions, Rt(ω) and Rt+1(ω), which can be converted from and to edge

perspective using the following formulas from [162].5

R(ω) =

∫ ω0ρ(ψ)dψ∫ 1

0ρ(ψ)dψ

(3.5)

ρ(ω) =R′(ω)

R′(1)(3.6)

Since the code construction splits the k syndrome nodes of largest degree out of

the total tk syndrome nodes (that is, a fraction of 1t) into pairs of equal or consecutive

degree, we partition Rt(ω) = Ft(ω) +Gt(ω), where Ft(ω) and Gt(ω) are nonnegative

polynomials, such that the maximum degree of the nonzero terms of Ft(ω) is less

than or equal to the minimum degree of the nonzero terms of Gt(ω). In this way,

Ft(ω) and Gt(ω) represent unnormalized node-perspective degree distributions of sets

of low and high degree syndrome nodes, respectively. Setting Gt(1) = 1t

(and hence

5In the node-perspective syndrome degree distribution polynomial R(ω), the coefficient of ωd is thefraction of syndrome nodes of source-syndrome degree d out of all syndrome nodes.


Ft(1) = 1 − 1t) means that the 1

tfraction of highest degree syndrome nodes are now

marked for splitting.

We further partition Gt(ω) = Get (ω) + Go

t (ω), where Get (ω) and Go

t (ω) are non-

negative polynomials, such that their nonzero terms have even and odd degree only,

respectively. Splitting the even degree syndrome nodes represented by Get (ω) results

in twice as many half degree nodes: 2Get (ω

12 ). Splitting the odd degree syndrome

nodes represented by Got (ω) results in equal numbers of nodes of just less than half

degree and just greater than half degree: ω−12Go

t (ω12 ) + ω

12Go

t (ω12 ).

Therefore, we can write the normalized node-perspective syndrome degree distri-

bution for the code with (t+ 1)k syndrome nodes as

Rt+1(ω) =Ft(ω) + 2Ge

t (ω12 ) + (ω−

12 + ω

12 )Go

t (ω12 )

Ft(1) + 2Get (1) + 2Go

t (1)(3.7)

=t

t+ 1

(Ft(ω) + 2Ge

t (ω12 ) + (ω−

12 + ω

12 )Go

t (ω12 )), (3.8)

since the normalization constant Ft(1) + 2Get (1) + 2Go

t (1) = Ft(1) + 2Gt(1) = 1 + 1t.

3.4 Rate-Adaptive LDPC Coding Experimental Results

Our experiments evaluate the coding performance of rate-adaptive LDPC codes and

the accuracy of the rate adaptation analysis using density evolution. The source

X = (X1, X2, . . . , Xn) and side information Y = (Y1, Y2, . . . , Yn) are both random

binary vectors. Within each vector, the elements are independent and equiprobable.

Between the vectors, the statistical dependence is given by Y = X + N modulo

2, where N is a random binary vector with independent elements equal to 1 with

probability ε. Thus, the relationship between X and Y is binary symmetric with

crossover probability ε = P{Xi 6= Yi}. The Slepian-Wolf bound is the conditional

entropy rate H(X|Y) = H(ε) = −ε log2(ε)− (1− ε) log2(1− ε) bit/bit.

We construct both regular and irregular rate-adaptive LDPC codes. In the regular

codes, all the source nodes have degree 3; that is, the source degree distribution is

λreg(ω) = ω2. The irregular codes have source degree distribution given by λirreg(ω) =

0.1317ω+ 0.2595ω2 + 0.1868ω6 + 0.1151ω7 + 0.0792ω18 + 0.2277ω20, a choice from the

optimized degree distributions in [14].


We consider codes of two lengths, n = 512 and 4096 bits. In both cases, we set

the encoded data increment size k = n128

; that is, k = 4 and 32 bits, respectively. In

this way, each sequence of codes achieves rates Radaptive ∈{

1128, 2

128, . . . , 1

}. Note that

it is not possible to construct the 512-bit irregular code because the maximal source

node degree of 21 exceeds the number of syndrome nodes k = 4 of the mother code.

Hence, we construct regular codes of lengths 512 and 4096 bits and irregular codes of

length 4096 bits only.

3.4.1 Performance Comparison with Other Codes

We compare the performance of the rate-adaptive LDPC codes with naıve syndrome-

deleted LDPC codes and punctured turbo codes [164] of code lengths, n = 512 and

4096 bits. Each sequence of syndrome-deleted LDPC codes are constructed from

a regular degree 3 LDPC mother code of rate 1. Lower rate codes at multiples of

code rate 1128

are obtained by syndrome deletion, as described in Section 3.2.1. Each

sequence of punctured turbo codes is generated using a pair of identical convolutional

encoders with generator matrix[

1 1+D+D3+D4

1+D3+D4

], as proposed for distributed source

coding in [7]. The puncturing of the parity bits is periodic so that the sequence takes

code rates in{

1128, 2

128, . . . , 1

}.

Fig. 3.6 plots the coding rates (averaged over 100 trials) required to compress

the source X with respect to the side information Y for varying conditional entropy

rate H(X|Y) = H(ε). The syndrome-deleted LDPC codes perform very poorly,

with coding rates far from the Slepian-Wolf bound when the entropy H(ε) is low.

The lower rate syndrome-deleted LDPC codes are never effective because syndrome

deletion severely degrades their factor graphs. The rate-adaptive LDPC codes and

punctured turbo codes, in contrast, compress close to the Slepian-Wolf bound for all

entropies H(ε). Note that the average performance of these codes is roughly the same

regardless of the length n = 512 or 4096 bits. The regular rate-adaptive LDPC codes

outperform the punctured turbo codes for H(ε) > 0.5, and the 4096-bit irregular

rate-adaptive LDPC codes outperform the punctured turbo codes for H(ε) > 0.2.

Fig. 3.7 plots the distributions of rate over the 100 trials of the experiments in

Fig. 3.6. For clarity, the rates are binned to the nearest multiple of 116

. Observe that

the 4096-bit codes have tighter distributions of their their rates than their respective


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)

Comparison of 512−bit Codes

SD LDPCPunctured turboRA LDPC (regular)SW bound

(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)ra

te (

bit/b

it)

Comparison of 4096−bit Codes

SD LDPCPunctured turboRA LDPC (regular)RA LDPC (irregular)SW bound

(b)

Figure 3.6: Comparison of coding performance of rate-adaptive (RA) LDPC codes withsyndrome-deleted (SD) LDPC codes and punctured turbo codes of lengths (a) n = 512 bitsand (b) n = 4096 bits.

512-bit codes. Moreover, the variances of the rate of the 4096-bit codes are similar,

regardless of the particular code or the entropy H(ε). So, even though the 4096-bit

codes have similar average performance as their 512-bit counterparts, the 4096-bit

codes are more predictable.

3.4.2 Performance of Rate Adaptation Analysis

We now evaluate the accuracy of density evolution in modeling the average mini-

mum coding rate. Our implementation is a Monte Carlo simulation using up to 214

samples [121].

Fig. 3.8 plots the modeled coding rates for both the regular and irregular rate-

adaptive LDPC codes of source degree distributions λreg(ω) and λirreg(ω), respectively.

The model very closely approximates the empirical results of the 4096-bit codes.


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

Rate Distribution at H(ε)=0.2 bit/bit

rate (bit/bit)

prob

abili

ty m

ass

func

tion

of r

ate

SD LDPC 512 bitsSD LDPC 4096 bitsPunctured turbo 512 bitsPunctured turbo 4096 bitsRA LDPC (regular) 512 bitsRA LDPC (regular) 4096 bitsRA LDPC (irregular) 4096 bits

(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


rate (bit/bit)pr

obab

ility

mas

s fu

nctio

n of

rat

e


(b)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


rate (bit/bit)

prob

abili

ty m

ass

func

tion

of r

ate


(c)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


rate (bit/bit)

prob

abili

ty m

ass

func

tion

of r

ate


(d)

Figure 3.7: Comparison of distributions of rate for rate-adaptive (RA) LDPC codes,syndrome-deleted (SD) LDPC codes and punctured turbo codes of lengths n = 512 and 4096bits, operating at entropies (a) H(ε) = 0.2 bit/bit, (b) H(ε) = 0.4 bit/bit, (c) H(ε) = 0.6bit/bit and (d) H(ε) = 0.8 bit/bit.


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)

DE Performance (regular)

RA LDPC 4096 bitsDE modelSW bound

(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)ra

te (

bit/b

it)

DE Performance (irregular)

RA LDPC 4096 bitsDE modelSW bound

(b)

Figure 3.8: Comparison of empirical coding performance of 4096-bit rate-adaptive (RA)LDPC codes and coding performance modeled by density evolution (DE) for (a) regularcodes and (b) irregular codes.

3.5 Summary

In this chapter, we show that rate-adaptive LDPC codes enable practical distributed

source coding with performance close to the Slepian-Wolf bound, that can be modeled

well by density evolution. We develop a construction for sequences of codes based

on the primitive of syndrome splitting. We derive the degree distributions of these

codes so that density evolution can be applied towards their analysis. Our experi-

mental results show that the rate-adaptive LDPC codes achieve performance close to

the Slepian-Wolf bound over the entire range of rates, and that density evolution is

accurate in modeling their empirical coding performance.

Chapter 4

Side Information Adaptation

This chapter considers distributed source coding in which each block of the source

at the encoder is associated with multiple candidates for side information at the

decoder, just one of which is statistically dependent on the source block. We argue

that the codec must compress and recover the source while simultaneously adapting

probabilities about which side information candidate best matches each source block.

We therefore call this topic side-information-adaptive distributed source coding.

In Section 4.1, we present distributed coding of random dot stereograms as a

motivating example and provide an overview of the operation of the side-information-

adaptive codec. Section 4.2 formalizes the statistical dependence between source and

side information as the block-candidate model and derives its conditional entropy

rate as the Slepian-Wolf bound. Section 4.3 describes the encoding of the source

into doping bits and cumulative syndrome bits, and the decoding as the sum-product

algorithm on a factor graph consisting of source, syndrome and side information

nodes. In Section 4.4, we analyze side information adaptation by transforming the

factor graph, deriving its degree distributions under varying rates of doping, and

applying a Monte Carlo simulation of density evolution. The experimental results in

Section 4.5 show that proposed codec performs close to the Slepian-Wolf bound when

the encoder sends an appropriate number of doping bits. Moreover, density evolution

accurately models the performance and is therefore used to design the doping rate.

30

CHAPTER 4. SIDE INFORMATION ADAPTATION 31

(a) (b)

Figure 4.1: Stereogram consisting of a pair of random dot images. To view the imagesstereoscopically, cross your eyes to the point that the images completely overlap. A squareshape in the center of the overlapped image appears to float out of the plane of the page.

4.1 Distributed Random Dot Stereogram Coding

Random dot stereograms represent perhaps the simplest possible model for stereo

images [91]. A random dot stereogram consists of a pair of statistically dependent

random dot images, like those shown in Fig. 4.1. By itself each image is devoid of

depth cues, but together they create a sensation of depth when viewed stereoscopi-

cally. Shapes appear to float in planes in front of or behind the actual image surface.

The optical illusion arises because statistically dependent regions of the pair of im-

ages are not necessarily located in the same position within their respective images,

as shown in the two superpositions in Fig. 4.2. Regions corresponding to floating

shapes are in fact shifted relative to each other and the perceived depth of the shape

scales with the amount of disparity. Colocated statistically dependent regions in the

pair of images, on the other hand, appear in the same plane as the surface.

The distributed source coding of the pair of images of the random dot stereogram is

a compelling problem because each image is by itself incompressible, but substantial

compression is possible by exploiting their joint statistics at the decoder. In the

lossless case, the total potential savings are available using asymmetric distributed


(a) (b)

Figure 4.2: Superpositions of the two random dot images of Fig. 4.1, colored red andcyan, respectively, with displacements of (a) 0 and (b) 8 pixels. The superposition withdisplacement 0 pixels reveals statistical dependence at the periphery, while that at 8 pixelsreveals statistical dependence in a central square region. These superposed images areanaglyphs and each could be viewed stereoscopically using red-cyan 3D glasses. As inFig. 4.1, a square shape in the center would appear to float out of the plane of the page.

source coding, in which one image is the source and the other is the side information.

Fig. 4.3 depicts three lossless asymmetric codecs that can be applied to distributed

random dot stereogram coding.

Baseline Codec

The baseline codec in Fig. 4.3(a) applies the rate-adaptive LDPC codes of Chapter 3

without modification. This means it ignores the disparity between the source and side

information and tries to exploit the statistical dependence between colocated pixels.

But, in regions of nonzero disparity, the colocated pixels are in fact statistically

independent. Consequently, the baseline system usually compresses the source very

poorly, and not at all when there is nonzero disparity throughout the random dot

stereogram.


LDPCEncoder

LDPCDecodersource

side information


encodedincrements

reconstruction

(a)

LDPCEncoder

LDPCDecodersource

side information


encodedincrements

reconstruction

Oracle

(b)

LDPCEncoder

LDPCDecodersource

side information


encodedincrements

reconstruction

Side InformationAdapter

(c)

Figure 4.3: Asymmetric distributed source coding of one random dot image with the otheras side information: (a) baseline, (b) oracle and (c) side-information-adaptive codecs.


Oracle Codec

The oracle codec in Fig. 4.3(b) builds on the baseline codec by adding a disparity

oracle to the decoder. The oracle realigns the regions of nonzero disparity in the side

information so that they are colocated with the matching regions in the source image.

In this way, the rate-adaptive LDPC codes exploit the true statistical dependence

between the pair of images, yielding better coding performance. But a fully informed

oracle is impossible because it requires knowledge of the source at the beginning of

the process of decoding the source.

Side-Information-Adaptive Codec

The side-information-adaptive codec in Fig. 4.3(c) provides a practical way for the

decoder to simultaneously learn the disparity and recover the source, at a coding rate

very competitive with that of the impractical oracle system. We replace the oracle

with a module called the side information adapter. Whereas the oracle realigns the

side information through the correct disparity map, the adapter considers multiple

side information candidates realigned through all possible disparity maps. The overall

decoder alternates between iterations of the LDPC decoder and the adapter, both of

which exchange statistical estimates of the source with each other. Each iteration of

LDPC decoding refines the source estimate using the increments of cumulative syn-

drome available so far. Each iteration of side information adaptation refines the source

estimate by computing the likelihoods of all the side information candidates. When

the decoding loop terminates after a fixed number of iterations, either the source is

recovered or the decoder requests a further increment of cumulative syndrome for use

in another decoding attempt.

Our original treatment of distributed random dot stereogram coding [209] casts

the iterative side-information-adaptive decoding algorithm as expectation maximiza-

tion [47]. The decoder’s goal is to recover the reconstruction as the maximum likeli-

hood estimate of the source given both the side information and the received incre-

ments of cumulative syndrome, treating the disparity as hidden variables. The expec-

tation step, which runs within the side information adapter, fixes the source estimate

and estimates the disparity. The maximization step fixes the disparity estimate and


estimates the source using one belief propagation iteration of LDPC decoding. The

remainder of this chapter takes a more abstract view of the side-information-adaptive

decoder and poses it in its entirety as the sum-product algorithm on a single factor

graph. The advantage of this approach is the side information adaptation analysis of

this algorithm using density evolution. We commence by formalizing the statistical

relationship between the source and side information.

4.2 Model for Source and Side Information

4.2.1 Definition of the Block-Candidate Model

Define the source X to be an equiprobable random binary vector of length n and the

side information Y to be a random binary matrix of dimension n× c,

X =

X1

X2

...

Xn

and Y =

Y1,1 Y1,2 · · · Y1,c

Y2,1 Y2,2 · · · Y2,c

......

. . ....

Yn,1 Yn,2 · · · Yn,c

. (4.1)

Assume that n is a multiple of the block size b. Then we further define blocks of X

and Y; namely, vectors x[i] of length b and matrices y[i] of dimension b× c,

x[i] =

X(i−1)b+1

X(i−1)b+2

...

Xib

and y[i] =

Y(i−1)b+1,1 Y(i−1)b+1,2 · · · Y(i−1)b+1,c

Y(i−1)b+2,1 Y(i−1)b+2,2 · · · Y(i−1)b+2,c

......

. . ....

Yib,1 Yib,2 · · · Yib,c

.

(4.2)

Finally, define a candidate y[i, j] to be the jth column of block y[i]; that is, y[i, j] =(Y(i−1)b+1,j, Y(i−1)b+2,j, · · · , Yib,j

), a vector of length b.

The statistics of the block-candidate model are illustrated in Fig. 4.4. The de-

pendence between X and Y is through the vector Z =(Z1, Z2, · · · , Zn

b

)of hidden

random variables, each of which is uniformly distributed over {1, 2, . . . , c}. Given

Zi = zi, the block x[i] has a binary symmetric relationship of crossover probability ε


c

b

source X side information Y

b

b

b

n

Figure 4.4: Block-candidate model of statistical dependence. The source X is an equiproba-ble binary vector of length n bits and the side information Y is a binary matrix of dimensionn× c. Binary values are shown as light/dark. For each block of b bits of X (among n

b suchnonoverlapping blocks), the corresponding b× c block of Y contains exactly one candidatedependent on X (shown color-coded.) The dependence is binary symmetric with crossoverprobability ε. All other candidates are independent of X.

with the candidate y[i, zi]. That is, y[i, zi] = x[i] +n[i] modulo 2, where n[i] is a ran-

dom binary vector with independent elements equal to 1 with probability ε. All other

candidates y[i, j 6= zi] in this block are equiprobable random vectors, independent of

x[i].

In the context of random dot stereograms, the blocks x[i] of X form a regular tiling

of the source image with each tile comprising b pixels. The candidates y[i, j] of a block

of Y represent the regions of the side information image that must be considered to

find the statistically dependent match with the source tile corresponding to x[i]. Thus,

the number of candidates c is the size of the search range in the side information

image for each tile in the source image. All three of the asymmetric distributed


source codecs of Fig. 4.3 recover the source X from its encoding in conjunction with

the side information Y, but they differ in their treatment of the hidden variables Z.

The baseline codec assumes that Z is fixed to some arbitrary value, the oracle codec

knows the true realization, and the side-information-adaptive codec infers it.

4.2.2 Slepian-Wolf Bound

The Slepian-Wolf bound for the block-candidate model is the conditional entropy rate

H(X|Y), which can be expressed as

H(X|Y) = H(X|Y,Z) +H(Z|Y)−H(Z|X,Y). (4.3)

The first term H(X|Y,Z) = H(ε) = −ε log2 ε − (1 − ε) log2(1 − ε) bit/bit, since

each block x[i] = y[i, zi] + n[i] modulo 2. Given Y and Z, the only randomness is

supplied by n[i], the random binary vectors with independent elements equal to 1

with probability ε.

The second term H(Z|Y) = H(Z) = 1bH(Zi) = 1

blog2 c bit/bit, since no informa-

tion about the hidden variables Z is revealed by the side information Y alone. Per

block of b bits, each variable Zi is uniformly distributed over c values.

The third term H(Z|X,Y) can be computed exactly by enumerating all joint

realizations of blocks (x[i],y[i]) = (x, y) along with their probabilities P{x, y} and

entropy terms H(Zi|x, y).

H(Z|X,Y) =1

bH(Zi|x[i],y[i]) (4.4)

=1

b

∑x,y

P{x, y}H(Zi|x, y) (4.5)

=1

b

∑y

P{y|x = 0}H(Zi|x = 0, y) (4.6)

The final equality sets x to 0 because the probability and entropy terms are unchanged

by flipping any bit in x and the colocated bits in the candidates of y. Since the term

H(Zi|x = 0, y) only depends on the number of bits in each candidate equal to 1, the

calculation is tractable for small values of b and c.


An alternative way to simplify the evaluation of H(Z|X,Y) is to assume that the

realization y|x = 0 is typical; that is, the statistically dependent candidate contains εb

bits of value 1 and all other candidates contain 12b bits of value 1 each. This assump-

tion of typicality is valid for large values of b, for which the asymptotic equipartition

property [46] deems∑

y typical

P{y|x = 0} ≈ 1. Approximating (4.6) gives

H(Z|X,Y) ≈ 1

bH(Zi|x = 0, y typical) (4.7)

=1

b(−pdep log2 pdep − (c− 1)pindep log2 pindep) , (4.8)

where pdep =wdep

wdep + (c− 1)windep

(4.9)

pindep =windep

wdep + (c− 1)windep

. (4.10)

Here, wdep = (1 − ε)(1−ε)bεεb and windep = (1 − ε) 12bε

12b are likelihoods weights of the

statistically dependent and independent candidates, respectively, being identified as

the statistically dependent candidate. In this way, the expression in (4.8) computes

the entropy of the index of the statistically dependent candidate.

Fig. 4.5 plots the Slepian-Wolf bounds for the block-candidate model as condi-

tional entropy rates H(X|Y), in exact form for tractable combinations of b and c

and approximated under the typicality assumption for b = 64. Note that the two

computations agree for the combination b = 64, c = 2. The Slepian-Wolf bound is

decreasing in block size b and increasing in both number of candidates c and H(ε).

4.3 Side-Information-Adaptive Codec

4.3.1 Encoder

The side-information-adaptive encoder’s objective is to code the source X to a rate

R > H(X|Y), the Slepian-Wolf bound. The bit stream comprises two segments. The

first segment, called the doping bits, is a sampling of the bits of X sent directly at

a fixed rate Rfixed bit/bit. The second segment consists of the cumulative syndrome

bits of a rate-adaptive LDPC code sent at a variable rate Radaptive bit/bit.


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

H(X

|Y)

(bit/

bit)

Slepian−Wolf Bounds with c=2

Exact H(X|Y), b=1Exact H(X|Y), b=4Exact H(X|Y), b=16Exact H(X|Y), b=64Approx H(X|Y), b=64H(ε)

(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)H

(X|Y

) (b

it/bi

t)


Exact H(X|Y), b=1Exact H(X|Y), b=4Exact H(X|Y), b=16Approx H(X|Y), b=64H(ε)

(b)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

H(X

|Y)

(bit/

bit)


Exact H(X|Y), b=1Exact H(X|Y), b=4Approx H(X|Y), b=64H(ε)

(c)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

H(X

|Y)

(bit/

bit)


Exact H(X|Y), b=1Approx H(X|Y), b=64H(ε)

(d)

Figure 4.5: Slepian-Wolf bounds for the block-candidate model, shown as conditional en-tropy rates H(X|Y) for number of candidates c equal to (a) 2, (b) 4, (c) 8 and (d) 16. Theexact form is shown for tractable values of block size b and the approximation (under thetypicality assumption) is shown for b = 64. Note that the exact and approximate formsagree for the combination b = 64, c = 2.


The purpose of doping is to initialize the side-information-adaptive decoder with

reliable information about X. The doping pattern is deterministic and regular, so

that each block x[i] contributes either bbRfixedc or dbRfixede doping bits. The rate-

adaptive LDPC code is constructed as described in Section 3.2.3 with code length n,

set to the length of X, and encoded data increment size k, set to a factor of n. Hence,

the variable rate Radaptive = t kn, where the code counter t is selected during operation

using a rate control feedback channel. For convenience, Rfixed is chosen in advance to

be a multiple of kn

as well.

4.3.2 Decoder

The role of the side-information-adaptive decoder is to recover the source X from

the doping bits and the cumulative syndrome bits so far received from the encoder

in conjunction with the block-candidate side information Y. We denote the received

vector of doping bits as (D1, D2, . . . , DnRfixed). Recall that taking consecutive sums of

the received cumulative syndrome bits modulo 2 produces a vector of syndrome bits,

which we write as the vector (S1, S2, . . . , SnRadaptive) in this section.

The decoder synthesizes all this information by applying the sum-product algo-

rithm on a factor graph structured like the one shown in Fig. 4.6. Each source node,

which represents a source bit, is connected to doping, syndrome and side information

nodes that bear some information about that source bit. The edges carry messages

that represent probabilities about the values of their attached source bits. The sum-

product algorithm iterates by message passing among the nodes so that ultimately

all the information is shared across the entire factor graph. The algorithm terminates

successfully when the source bit estimates, when thresholded, are consistent with the

vector of syndrome bits. If this condition is not reached within a maximum number of

iterations, the decoder increments Radaptive by requesting more cumulative syndrome

bits from the encoder. Then the vector of syndrome bits is regenerated and the sum-

product algorithm is applied to a new factor graph that includes the new syndrome

bits. This process repeats until successful termination.

The nodes in the graph accept input messages from and produce output messages

for their neighbors. These messages represent probabilities that the connected source


side informationnodes

S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

S12

S14

S16

S11

S13

S15

sourcenodes

syndromenodes

y[1]

y[3]

y[2]

y[4]

D2

D1

Figure 4.6: Factor graph for side-information-adaptive decoder with parameters: codelength n = 32, block size b = 8, number of candidates c = 4, Rfixed = 1

16 and Radaptive = 12 .

The associated rate-adaptive LDPC code is regular with edge-perspective source and syn-drome degree distributions λt(ω) = ω2 and ρt(ω) = ω5, respectively, where the code countert = n

kRadaptive.

bits are 0, and are denoted by vectors (pin1 , p

in2 , . . . , p

ind ) and (pout

1 , pout2 , . . . , pout

d ), re-

spectively, where d is the degree of the node. By the sum-product rule, each output

message poutu is a function of all input messages except pin

u , the input message on the

same edge. Each source node additionally produces an estimate pest that its bit is 0,

based on all the input messages. In the rest of this section, we detail the computation

rules of the nodes or combination of nodes shown in Fig. 4.7.


p1in

p1out

p2in

p2out

p3in

p3out

pdin

pdout

pest

(a)

Dp1

in

p1out

p2in

p2out

p3in

p3out

pdin

pdout

pest

(b)

S

p1in

p1out

p2in

p2out

pdin

pdout

(c)

p1in

p1out

p2in

p2out

pbin

pbout

Y1,1 Y1,2 … Y1,c

Y2,1 Y2,2 … Y2,c

Yb,1 Yb,2 … Yb,c

… … …y[i]

(d)

Figure 4.7: Factor graph node combinations. Input and output messages and source bitestimate (if applicable) of (a) source node unattached to doping node, (b) source nodeattached to doping node, (c) syndrome node and (d) side information node.

Source Node Unattached to Doping Node

Fig. 4.7(a) shows a source node unattached to a doping node. There are two outcomes

for the source bit random variable: it is 0 with likelihood weight∏d

v=1 pinv or 1 with

likelihood weight∏d

v=1(1− pinv ). Consequently,

pest =

d∏v=1

pinv

d∏v=1

pinv +

d∏v=1

(1− pinv )

. (4.11)


Ignoring the input message pinu , the weights are

∏v 6=u p

inv and

∏v 6=u(1− pin

v ), so

poutu =

∏v 6=u

pinv∏

v 6=u

pinv +

∏v 6=u

(1− pinv ). (4.12)

Source Node Attached to Doping Node

Fig. 4.7(b) shows a source node attached to a doping node of binary value D. Recall

that the doping bit specifies the value of the source bit, so the source bit estimate

and the output messages are independent of the input messages,

pest = poutu = 1−D. (4.13)

Syndrome Node

Fig. 4.7(c) shows a syndrome node of binary value S. Since the connected source

bits have modulo 2 sum equal to S, the output message poutu is the probability that

the modulo 2 sum of all the other connected source bits is equal to S. We argue by

mathematical induction on d that

poutu =

1

2+

1− 2S

2

∏v 6=u

(2pinv − 1). (4.14)

Side Information Node

Fig. 4.7(d) shows a side information node of value y[i] consisting of b× c bits, all of

which are labeled in Fig. 4.7(d) omitting the block index i for notational simplicity.

Just as for the other types of node, the computation of the output message poutu

depends on all input messages except pinu . But since the source bits and the bits of

the dependent candidate are related through a crossover probability ε, we define the

noisy input probability of a source bit being 0 by

pnoisy-inv = (1− ε)pin

v + ε(1− pinv ). (4.15)


In computing poutu , the likelihood weight of the candidate y[i, j] being the statistically

dependent candidate equals the product of the likelihoods of that candidate’s b − 1

bits excluding Yu,j,

wu,j =∏v 6=u

(1[Yv,j=0]p

noisy-inv + 1[Yv,j=1](1− pnoisy-in

v )), (4.16)

where 1[.] is the indicator function. We finally marginalize poutu as the normalized sum

of weights for which Yu,j is 0, passed through crossover probability ε,

pclean-outu =

c∑j=1

1[Yu,j=0]wu,j

c∑j=1

wu,j

(4.17)

poutu = (1− ε)pclean-out

u + ε(1− pclean-outu ). (4.18)

4.4 Analysis of Side Information Adaptation

We use density evolution to determine whether the sum-product algorithm converges

on the proposed factor graph. Our approach first transforms the factor graph into

a simpler one that is equivalent with respect to convergence. Next we derive degree

distributions for this graph. Finally, we describe a Monte Carlo simulation of density

evolution for the side-information-adaptive decoder.

4.4.1 Factor Graph Transformation

The convergence of the sum-product algorithm is invariant under manipulations to

the source, side information, syndrome and doping bits and the factor graph itself as

long as the messages passed along the edges are preserved up to relabeling.

The first simplification is to reorder the candidates within each side information

block so that statistically dependent candidate is in the first position y[i, 1]. This

shuffling has no effect on the messages.

We then replace each side information candidate y[i, j] with the modulo 2 sum of

itself and its corresponding source block x[i], and set all the source bits, syndrome


bits and doping bits to 0. The values of the messages would be unchanged if we would

relabel each message to stand for the probability that the attached source bit (which

is now 0) is equal to the original value of that source bit.

Finally, observe that any source node attached to a doping node always outputs

deterministic messages equal to 1, since the doping bit D is set to 0 in (4.13). We

therefore remove all instances of this node combination along with all their attached

edges from the factor graph. In total, a fraction Rfixed of the source nodes are removed.

Although some edges are removed at some syndrome nodes, no change is required to

the syndrome node decoding rule because ignoring input messages pinv = 1 does not

change the term∏

v 6=u(2pinv − 1) in (4.14). In contrast, side information nodes with

edges removed must substitute the missing input messages with 1 in (4.15) to (4.18).

Applying these three manipulations to the factor graph of Fig. 4.6 produces the

simpler factor graph, equivalent with respect to convergence, shown in Fig. 4.8, in

which the values 0 and 1 are denoted light and dark, respectively. The syndrome nodes

all have value 0, which is consistent with the source bits all being 0 as well. Only the

side information candidates in the first position y[i, 1] are statistically dependent with

respect to the source bits. In particular, the bits of y[i, 1] are independently equal to

0 with probability 1− ε, while the bits of y[i, j 6= 1] are independently equiprobable.

Note also the absence of combinations of source nodes linked to doping nodes.

4.4.2 Derivation of Degree Distributions

Density evolution runs, not on a factor graph itself, but using degree distributions

of that factor graph. Recall that degree distributions exist in two equivalent forms,

edge-perspective and node-perspective. In an edge-perspective degree distribution

polynomial, the coefficient of ωd−1 is the fraction of edges connected to a certain type

of node of degree d out of all edges connected to nodes of that type. In a node-

perspective degree distribution polynomial, the coefficient of ωd is the fraction of a

certain type of node of degree d out of all nodes of that type.

In total, we derive twelve degree distributions, six for each of two different graphs.

The source, syndrome and side information degree distributions of the factor graph

before transformation (like the one in Fig. 4.6) are respectively labeled λt(ω), ρt(ω)

and βt(ω) in edge perspective and Lt(ω), Rt(ω) and Bt(ω) in node perspective, where



S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

S12

S14

S16

S11

S13

S15

sourcenodes

syndromenodes

y[1]

y[3]

y[2]

y[4]

Figure 4.8: Transformed factor graph equivalent to that of Fig. 4.6 in terms of convergence.After doping, the edge-perspective source and syndrome degree distributions of the asso-ciated rate-adaptive LDPC code are λ∗t (ω) = ω2 and ρ∗t (ω) = 2

45ω3 + 2

9ω4 + 11

15ω5. The

edge-perspective side information degree distribution is β∗t (ω) = 715ω

6 + 815ω

7.

the code counter t = nkRadaptive. Their expected counterparts in the factor graph after

transformation (like the one in Fig. 4.8) are denoted λ∗t (ω), ρ∗t (ω) and β∗t (ω) in edge

perspective and L∗t (ω), R∗t (ω) and B∗t (ω) in node perspective.

For source degree distributions λt(ω), Lt(ω), λ∗t (ω) and L∗t (ω), we count the source-

syndrome edges, but neither the source-side-information nor source-doping edges.

In this way, λt(ω), ρt(ω), Lt(ω) and Rt(ω) are consistent with the corresponding

definitions for rate-adaptive LDPC codes.


Source Degree Distributions

Recall from (3.1) that the edge-perspective source degree distribution λt(ω) is in-

variant with respect to t and equal to the designed source degree distribution λ(ω).

During the factor graph transformation, a fraction Rfixed of the source nodes are se-

lected for removal regardless of their degrees. Therefore, the expected source degree

distributions are preserved,

λ∗t (ω) = λt(ω) = λ(ω), (4.19)

L∗t (ω) = Lt(ω) =

∫ ω0λ(ψ)dψ∫ 1

0λ(ψ)dψ

, (4.20)

using the edge-perspective to node-perspective conversion formula in [162].

Syndrome Degree Distributions

The edge-perspective syndrome degree distribution ρt(ω) is obtained in (3.2) to (3.8)

by performing an inductive process on the node-perspective syndrome degree distri-

bution Rt(ω). We derive R∗t (ω) from Rt(ω) as described below, and obtain ρ∗t (ω) by

differentiating and normalizing R∗t (ω), using a formula analogous to (3.6).

Notice that the factor graph transformation, by removing a fraction Rfixed of the

source nodes regardless of their degrees, removes the same fraction of source-syndrome

edges in expectation. From the perspective of a syndrome node of original degree

d, each edge is retained independently with probability 1 − Rfixed. Consequently,

the chance that it has degree d∗ after factor graph transformation is the binomial

probability(dd∗

)(1−Rfixed)d

∗(Rfixed)d−d

∗. So if the node-perspective syndrome degree

distribution before transformation is expressed as Rt(ω) =∑dmax

d=1 Adωd, then after

transformation the expected node-perspective syndrome degree distribution is given

by

R∗t (ω) =dmax∑d=1

Ad1− (Rfixed)d

d∑d∗=1

(d

d∗

)(1−Rfixed)d

∗(Rfixed)d−d

∗ωd∗, (4.21)

where the normalization factor 11−(Rfixed)d accounts for the fact that degree d∗ = 0

syndrome nodes are not included in the degree distribution.


Side Information Degree Distributions

In the factor graph before transformation, all side information nodes have degree b,

so the edge- and node-perspective side information degree distributions are

βt(ω) = ωb−1, (4.22)

Bt(ω) = ωb. (4.23)

Since the doping pattern is deterministic and regular at rate Rfixed, each side in-

formation node retains either b∗ or b∗ + 1 edges in the transformed graph, where

b∗ = bb(1 − Rfixed)c. With fractional part A = b(1 − Rfixed) − b∗, the node- and

edge-perspective side information degree distributions after transformation are

B∗t (ω) = (1− A)ωb∗

+ Aωb∗+1, (4.24)

β∗t (ω) =(1− A)b∗

b∗ + Aωb∗−1 +

A(b∗ + 1)

b∗ + Aωb∗, (4.25)

where β∗t (ω) is obtained from B∗t (ω) by differentiation and normalization.

4.4.3 Monte Carlo Simulation of Density Evolution

We now use densities to represent the distributions of messages passed among classes

of nodes. The source-to-side-information, source-to-syndrome, syndrome-to-source

and side-information-to-source densities are denoted Qso-si, Qso-syn, Qsyn-so and Qsi-so,

respectively. Another density Qsource captures the distribution of source bit estimates.

Fig. 4.9 depicts a schematic of the density evolution process. The message den-

sities are passed among three stochastic nodes that represent the side information,

source and syndrome nodes. Inside the nodes are written the probabilities that the

values at those positions are 0. Observe that the source and syndrome stochastic

nodes are deterministically 0 and only the elements of the candidate in the first posi-

tion of the side information stochastic node are biased towards 0, in accordance with

the transformation in Section 4.4.1. Fig. 4.9 also shows the edge-perspective degree

distributions of the transformed factor graph beside the stochastic nodes. Since ev-

ery source node connects to exactly one side information node, the edge-perspective

source degree distribution with respect to the side information nodes is 1.


½…½1-ε

½…½1-ε½…½1-ε

… … … 1 1βt*(ω) ρt*(ω)λt*(ω)1

Qso-si

Qsyn-soQsi-so

Qso-synQsource

Figure 4.9: Density evolution for side-information-adaptive decoder. The three stochasticnodes are representatives of the side information, source and syndrome nodes, respectively,of a transformed factor graph, like the one in Fig. 4.8. The quantities inside the nodes arethe probabilities that the values at those positions are 0. Beside the nodes are written theexpected edge-perspective degree distributions of the transformed factor graph. The arrowlabels Qso-si, Qso-syn, Qsyn-so and Qsi-so are densities of messages passed in the transformedfactor graph, and Qsource is the density of source bit estimates.

During density evolution, each message density is stochastically updated as a

function of the values and degree distributions associated with the stochastic node

from which it originates and the other message densities that arrive at that stochastic

node. After a fixed number of iterations of evolution, Qsource is evaluated. The sum-

product algorithm is deemed to converge for the factor graph in question if and only if

the density of source bit estimates Qsource, after thresholding, converges to the source

bit value 0.

The rest of this section provides the stochastic update rules for the densities in

a Monte Carlo simulation of density evolution. For the simulation, the each of the

densities Qsource, Qso-si, Qso-syn, Qsyn-so and Qsi-so is defined to be a set of samples q,

each of which is a probability that its associated source bit is 0. At initialization, all

samples q of all densities are set to 12.

Source Bit Estimate Density

To compute each sample qout of Qsource, let a set Qin consist of 1 sample drawn

randomly from Qsi-so and δ samples drawn randomly from Qsyn-so. The random


degree δ is drawn equal to d with probability equal to the coefficient of ωd in node-

perspective L∗t (ω), since there is one actual source bit estimate per node. Then,

according to (4.11),

qout =

∏qin∈Qin

qin

∏qin∈Qin

qin +∏

qin∈Qin

(1− qin). (4.26)

Source-to-Side-Information Message Density

To compute each updated sample qout of Qso-si, let a set Qin consist of δ samples drawn

randomly from Qsyn-so. The random degree δ is drawn equal to d with probability

equal to the coefficient of ωd in node-perspective L∗t (ω), since there is one actual

output message per node. Using (4.12), the update formula is the same as (4.26).

Source-to-Syndrome Message Density

To compute each updated sample qout of Qso-syn, let a set Qin consist of 1 sample

drawn randomly from Qsi-so and δ − 1 samples drawn randomly from Qsyn-so. The

random degree δ is drawn equal to d with probability equal to the coefficient of ωd−1

in edge-perspective λ∗t (ω), since there is one actual output message per edge. Using

(4.12), the update formula is the same as (4.26).

Syndrome-to-Source Message Density

To compute each updated sample qout of Qsyn-so, let a set Qin consist of δ− 1 samples

drawn randomly from Qso-syn. The random degree δ is drawn equal to d with prob-

ability equal to the coefficient of ωd−1 in edge-perspective ρ∗t (ω), since there is one

actual output message per edge. The syndrome value is deterministically 0. Then,

according to (4.14),

qout =1

2+

1

2

∏qin∈Qin

(2qin − 1). (4.27)


Side-Information-to-Source Message Density

To compute each updated sample qout of Qsi-so, let a set{qinv

}b−1

v=1consist of δ − 1

samples drawn randomly from Qso-si and b − δ samples equal to 1. The random

degree δ is drawn equal to d with probability equal to the coefficient of ωd−1 in edge-

perspective β∗t (ω), since there is one actual output message per edge. The samples set

to 1 substitute for the messages on edges removed during factor graph transformation

due to doping.

For each qout, create also a realization of a b × c block of side information from

the joint distribution induced by the side information stochastic node. That is, each

element Yv,j is independently biased towards 0, with probability 1 − ε if j = 1 or

probability 12

if j 6= 1.

Generate sets{qnoisy-inv

}b−1

v=1and {wj}cj=1, before finally updating sample qout, fol-

lowing (4.15) to (4.18).

qnoisy-inv = (1− ε)qin

v + ε(1− qinv ) (4.28)

wj =b−1∏v=1

(1[Yv,j=0]q

noisy-inv + 1[Yv,j=1](1− qnoisy-in

v ))

(4.29)

qclean-out =

c∑j=1

1[Yb,j=0]wj

c∑j=1

wj

(4.30)

qout = (1− ε)qclean-out + ε(1− qclean-out) (4.31)

4.5 Side-Information-Adaptive Coding Experimental Results

We evaluate the coding performance of the side-information-adaptive codec and the

accuracy of its analysis using density evolution, for binary source X and binary side

information Y under a variety of settings of the block-candidate model. Our key

findings are that good choices for the doping rate Rfixed are required to achieve com-

pression close to the Slepian-Wolf bound, and that the density evolution model suc-

cessfully makes those choices.


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)

Performance without Doping b=64, c=16

SIA codecDE modelOracle codecSW boundH(ε)Doping rate

(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)ra

te (

bit/b

it)

Performance with Doping b=64, c=16

SIA codecDE modelOracle codecSW boundH(ε)Doping rate

(b)

Figure 4.10: Comparison of performance of side-information-adaptive (SIA) codec withdoping rate Rfixed (a) set to 0 and (b) set optimally using density evolution (DE).

The side-information-adaptive codec, in all our experiments, uses rate-adaptive

LDPC codes of length n = 4096 bits, encoded data increment size k = 32 bits, and

regular source degree distribution λreg(ω) = ω2. Hence, Radaptive ∈{

1128, 2

128, . . . , 1

}.

For convenience, we allow Rfixed ∈{

0, 1128, 2

128, . . . , 1

8

}. Our side information adapta-

tion analysis is implemented as a Monte Carlo simulation of density evolution using

up to 214 samples.

4.5.1 Performance with and without Doping

Fig. 4.10 illustrates the importance of the doping rate Rfixed for the setting of the

block-candidate model with block size b = 64 bits and number of candidates c = 16.

We plot the overall coding rates (each averaged over 100 trials) of the side-information-

adaptive codec and the corresponding oracle codec as we vary the entropy H(ε) of the

noise between X and the statistically dependent candidates of Y. We also plot the

side-information-adaptive codec’s Slepian-Wolf bound H(X|Y) and its performance

modeled according to density evolution. Note that the oracle decoder knows a priori


which candidates of Y are the ones statistically dependent on X, and its own Slepian-

Wolf bound is H(ε).

In Fig. 4.10(a), settingRfixed = 0, the performance of the side-information-adaptive

codec without doping is far from both the performance of the oracle codec and the

Slepian-Wolf bound, since the side-information-adaptive decoder is not initialized

with sufficient reliable information about X. Remarkably, density evolution models

the poor empirical performance with reasonably good accuracy.

Increasing the doping rate Rfixed initializes the decoder better, but increasing it too

much penalizes the overall side-information-adaptive codec rate. We therefore search

through all values Rfixed ∈{

0, 1128, 2

128, . . . , 1

8

}and let density evolution determine

which minimizes the coding rate for each H(ε). Fig. 4.10(b) shows the performance

with these optimal doping rates Rfixed. The side-information-adaptive codec operates

close to both the Slepian-Wolf bound and the oracle codec’s performance, and is

approximated by density evolution even better than when Rfixed = 0 in Fig. 4.10(a).

4.5.2 Performance under Different Block-Candidate Model Settings

We vary the block size b and fix the number of candidates c = 2 of the block-candidate

model in Fig. 4.11 and fix the block size b = 64 bits and vary the number of candidates

c in Fig. 4.12. For each setting, we find the optimal doping rates Rfixed and plot the

side-information-adaptive codec’s empirical performance and performance modeled

by density evolution. The corresponding Slepian-Wolf bounds are computed exactly

in Fig. 4.11 and approximately in Fig. 4.12 using the derivations in Section 4.2.2.

These figures demonstrate that, with optimal doping, the performance of the side-

information-adaptive codec is close to the Slepian-Wolf bound and is modeled well

using density evolution, under a variety of settings. As the block size increases while

the number of candidates is fixed, the doping rate decreases, and as the number of

candidates increases while the block size is fixed, the doping rate increases. Large

and small H(ε) also require greater doping rates than intermediate values of H(ε).

Note that, at H(ε) = 0.9 bit/bit, the coding rate saturates at 1 bit/bit, so Rfixed = 0

suffices.


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)


SIA codecDE modelSW boundH(ε)Doping rate

(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)ra

te (

bit/b

it)



(b)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)



(c)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)



(d)

Figure 4.11: Coding performance of side-information-adaptive (SIA) codec and performancemodeled according to density evolution (DE), with optimal doping rate Rfixed, under block-candidate model with number of candidates c fixed to 2 and different block sizes b equal to(a) 8, (b) 16, (c) 32 and (d) 64 bits.


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)



(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)ra

te (

bit/b

it)



(b)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)



(c)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)



(d)

Figure 4.12: Coding performance of side-information-adaptive (SIA) codec and performancemodeled according to density evolution (DE), with optimal doping rate Rfixed, under block-candidate model with block size b fixed to 64 bits and different numbers of candidates cequal to (a) 2, (b) 4, (c) 8 and (d) 16.


4.6 Summary

This chapter covers side-information-adaptive distributed source coding, in which

each block of the source is statistically dependent on just one of several candidates

of the side information. We provide a motivation for and an overview of this work

in our discussion on distributed source coding of random dot stereograms. The sta-

tistical relationship between the source and side information is formalized in terms

of the block-candidate model, which admits the derivation of the conditional en-

tropy rate as the Slepian-Wolf bound. The encoder of the proposed side-information-

adaptive codec encodes the source into two segments: a fixed rate of doping bits

and a variable rate of cumulative syndrome bits (using rate-adaptive LDPC codes.)

The decoder recovers the source with reference to the side information by apply-

ing the sum-product algorithm to a factor graph consisting of source, syndrome and

side information nodes. We analyze the side information adaptation of this codec

by first transforming its factor graph into a simpler one that is equivalent with re-

spect to convergence. After obtaining degree distributions for the transformed graph,

we apply a Monte Carlo simulation of density evolution. Our experimental results

show that the side-information-adaptive codec compresses the source at rates close

to the Slepian-Wolf bound under a variety of settings of the block-candidate model

as long as the doping rate is set optimally. We use density evolution to find the best

choices of doping rate, since it accurately models the empirical performance of the

side-information-adaptive codec.

Chapter 5

Multilevel Side Information

Adaptation

This chapter extends the binary techniques of Chapter 4 to the adaptive distributed

source coding of a multilevel source with respect to multilevel side information.

In Section 5.1, we spell out the new challenges in both the coding and the anal-

ysis by density evolution. Section 5.2 extends the block-candidate statistical model

to the multilevel case and derives the Slepian-Wolf bound. In Section 5.3, we de-

scribe the whole symbol encoder, which applies either a binary or Gray symbol-to-bit

mapping before using the side-information-adaptive encoder, and the whole symbol

decoder, which augments the factor graph of the side-information-adaptive decoder

with symbol and mapping nodes. In Section 5.4, we analyze the multilevel extension

by transforming the augmented factor graph, deriving its degree distributions under

varying rates of doping, and applying a Monte Carlo simulation of density evolution.

The experimental results in Section 5.5 demonstrate that the codec’s performance is

close to the Slepian-Wolf bound and well modeled by the density evolution analy-

sis, for both binary and Gray mappings. We also show that the codec using Gray

mapping usually requires a lower doping rate than the one using binary mapping.

57

CHAPTER 5. MULTILEVEL SIDE INFORMATION ADAPTATION 58

5.1 Challenges of Extension to Multilevel Coding

Multilevel source and side information introduce new challenges for adaptive dis-

tributed source coding and its analysis. The codec must now apply whole symbol

coding to perform efficient learning of the hidden variables that relate the source

and side information. The density evolution analysis, in order to capture symbol

processing, requires certain symmetry conditions to hold.

Whole Symbol Coding

One way to apply a binary codec to multilevel symbols is bit-plane-by-bit-plane,

making sure to exploit the redundancy among bit planes. This approach works for

rate-adaptive distributed source coding if each bit plane decoded becomes additional

side information at the decoder for the coding of subsequent bit planes [7, 38]. But

the bit-plane-by-bit-plane extension does not work well for side-information adaptive

distributed source coding because each source bit plane contributes different informa-

tion about which candidates of the side information are the statistically dependent

ones. Fig. 5.1 shows an example for source and side information blocks, x[i] and

y[i], respectively, in which each symbol is represented by 2 bits. Either the first or

second bit plane alone suggests that either candidate y[i, 2] or y[i, 3], respectively,

is the closest match to x[i]. But both bit planes together reveal that y[i, 1] is in

fact the matching candidate. We therefore require whole symbol coding instead of

bit-plane-by-bit-plane coding.

Symmetry Conditions for Density Evolution Analysis

The method of density evolution makes the simplifying assumption that the source

symbols are all 0 without loss of generality. This assumption is valid if the probability

of a joint realization of source and side information blocks is unchanged if any bit of

any symbol in the source block is flipped and the colocated bits in the side information

candidates are also flipped.

In the binary block-candidate model, we therefore require each source block to

have a binary symmetric statistical dependence with its matching side information

candidate. We now generalize the symmetry conditions for multilevel symbols. The


source blockx[i]

side information blocky[i]

0000000000000000

01 01 1000 01 1000 01 1010 01 1000 01 1000 01 1000 01 1000 01 10

Figure 5.1: Comparison of whole symbol and bit-plane-by-bit-plane statistics. The sourceblock x[i] matches candidates y[i, 1], y[i, 2] and y[i, 3] in 6, 0 and 0 symbols, respectively.But the number of corresponding matches in the first bit plane are 7, 8 and 0 bits, and inthe second bit plane are 7, 0 and 8 bits, reading bit planes from the left.

symbol values must form a circular ordering, so that all values are in equivalent

positions. The distribution of a side information symbol that is statistically dependent

on its corresponding source symbol must be symmetric around the value of that source

symbol, so that any circular ordering is equivalent to the ordering reversed. Finally,

the symbol values must map to bit representations in such a way that, when the bits

of any subset of bit planes are flipped, the new circular ordering is isomorphic to the

original ordering. This isomorphism condition ensures that the symbol values retain

their relative positions after bit flipping.

The isomorphism condition holds for both binary and Gray mappings of 2-bit

symbols. Fig. 5.2(a) shows the binary mapping, and Fig. 5.2(b) to (d) show the

same mapping with least significant bit (LSB) plane flipped, most significant bit

(MSB) plane flipped and both bit planes flipped. All the resulting binary orderings,

depicted by the arrows, are isomorphic. The same is demonstrated for the 2-bit Gray

mapping in Fig. 5.2(e) to (h). But this property does not extend to 3-bit symbols.

Fig. 5.3(a) and (c) show the binary and Gray mappings and Fig. 5.3(b) and (d) show

the respective mappings with LSB plane flipped. In both cases, the new ordering

is not isomorphic to the original. In fact, it can be shown that the isomorphism

condition only holds for 2-bit symbols.


00

10

0111

(a)

01

11

0010

(b)

10

00

1101

(c)

11

01

1000

(d)

00

11

0110

(e)

01

10

0011

(f)

10

01

1100

(g)

11

00

1001

(h)

Figure 5.2: Mappings of 2-bit symbols: (a) binary, (b) binary with LSB plane flipped,(c) binary with MSB plane flipped, (d) binary with both bit planes flipped, (e) Gray,(f) Gray with LSB plane flipped, (g) Gray with MSB plane flipped and (h) Gray with bothbit planes flipped. The binary orderings in (a) to (d) are isomorphic and so are the Grayorderings in (e) to (h).

000

100

010110

011

001111

101

(a)

001

101

011111

010

000110

100

(b)

000

110010

001100

111

011101

(c)

001

111

010100

011

000101

110

(d)

Figure 5.3: Mappings of 3-bit symbols: (a) binary, (b) binary with LSB plane flipped,(c) Gray and (d) Gray with LSB plane flipped. The binary orderings in (a) and (b) are notisomorphic, nor are the Gray orderings in (c) and (d).

Although these restrictive symmetry conditions are necessary for analysis of the

codec, they are not required for the codec itself. Hence, in the following section, we

extend the definition of the block-candidate model to multilevel symbols of 2 or more

bits.


5.2 Model for Multilevel Source and Side Information

5.2.1 Multilevel Extension of the Block-Candidate Model

The multilevel block-candidate model has much in common with its binary counter-

part defined in Section 4.2.1. The source X is a random vector of length n and the side

information Y is a random matrix of dimension n× c. They are composed of blocks

x[i] of length b and y[i] of dimension b× c, respectively, where b is a factor of n. The

statistical dependence between X and Y is through the vector Z =(Z1, Z2, · · · , Zn

b

)of hidden random variables, each of which is uniformly distributed over {1, 2, . . . , c}.

In the multilevel extension, X consists of symbols of 2m levels, uniformly dis-

tributed over {0, 1, . . . , 2m − 1}. Given Zi = zi, the candidate y[i, zi] is statistically

dependent on the block x[i] according to y[i, zi] = x[i] + n[i] modulo 2m, where the

random vector n[i] has independent symbols equal to l ∈ {0, 1, . . . , 2m−1} with prob-

ability εl. The symbols of all other candidates y[i, j 6= zi] in this block are uniformly

distributed over {0, 1, . . . , 2m − 1} and independent of x[i].

For symmetry and convenience, we usually constrain the statistical dependence

between x[i] and y[i, zi] with a single parameter σ such that

εl =ζl

ζ0 + ζ1 + · · ·+ ζ2m−1

, (5.1)

where ζl =

exp(− l2

2σ2

), if 0 ≤ l < 2m−1

exp(− (2m−l)2

2σ2

), if 2m−1 ≤ l < 2m

, (5.2)

and define the entropy H(σ) = H(ε0, ε1, . . . , ε2m−1) = −2m−1∑l=0

εl log2 εl bit/symbol.

5.2.2 Slepian-Wolf Bound

The Slepian-Wolf bound for the multilevel block-candidate model is the conditional

entropy rate H(X|Y) and is derived using similar arguments as in Section 4.2.2.

The results (4.3) to (4.5) are the same with the exception that H(X|Y,Z) = H(σ),

instead of H(ε). Assuming the symmetry conditions of Section 5.1, we replace the

source symbols with 0 and thereby simplify H(Z|X,Y) to the expression in (4.6).


If we assume typicality in addition, the statistically dependent candidates contain

εlb symbols of value l and all other candidates contain b2m symbols of value l, for all

l ∈ {0, 1, . . . , 2m−1}. We can thus approximate H(Z|X,Y) using (4.7) to (4.10) with

different likelihood weights wdep =2m−1∏l=0

εεlbl and windep =2m−1∏l=0

εb

2m

l .

Fig. 5.4 plots the Slepian-Wolf bounds for the multilevel block-candidate model

with m = 2 as conditional entropy rates H(X|Y), in exact form for tractable combi-

nations of b and c and approximated under the typicality assumption for b = 64.

5.3 Multilevel Side-Information-Adaptive Codec

5.3.1 Whole Symbol Encoder

The multilevel encoder first maps the source X from n symbols of 2m levels into

n′ = mn bits, using an m-bit binary or Gray mapping. Then it codes the n′ bits

into two segments, like the encoder in Section 4.3.1. The doping bits are sampled

deterministically and regularly at a fixed rate Rfixed bit/symbol. The cumulative

syndrome bits are produced at a variable rate Rvariable bit/symbol by a rate-adaptive

LDPC code of code length n′ and encoded data increment size k, which is a factor

of n. In this way, Rvariable = t kn, where t is the code counter, is consistent with its

definition in Chapters 3 and 4. As before, Rfixed is chosen in advance also to be a

multiple of kn, for convenience.

5.3.2 Whole Symbol Decoder

The multilevel decoder recovers the source X from different information about its bit

representation and its symbol values. The doping bits (D1, D2, . . . , Dn′Rfixed) and the

syndrome bits (S1, S2, . . . , Sn′Radaptive), obtained from the cumulative syndrome bits

so far received, pertain to the bit representation of X. In contrast, the multilevel side

information Y, contains information about the symbol values of X.

The decoder synthesizes all the bit and symbol information by applying the sum-

product algorithm on a factor graph structured like the one shown in Fig. 5.5. Like

the factor graph in Fig. 4.6, each source node represents a bit and is connected

to doping and syndrome nodes that bear information about that bit. Unlike the


0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

H(X

|Y)

(bit/

sym

bol)

Slepian−Wolf Bounds with m=2, c=2

Exact H(X|Y), b=1Exact H(X|Y), b=4Exact H(X|Y), b=16Exact H(X|Y), b=64Approx H(X|Y), b=64H(σ)

(a)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)H

(X|Y

) (b

it/sy

mbo

l)


Exact H(X|Y), b=1Exact H(X|Y), b=4Approx H(X|Y), b=64H(σ)

(b)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

H(X

|Y)

(bit/

sym

bol)


Exact H(X|Y), b=1Approx H(X|Y), b=64H(σ)

(c)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

H(X

|Y)

(bit/

sym

bol)


Approx H(X|Y), b=64H(σ)

(d)

Figure 5.4: Slepian-Wolf bounds for multilevel block-candidate model with m = 2, shown asconditional entropy rates H(X|Y) for number of candidates c equal to (a) 2, (b) 4, (c) 8 and(d) 16. The exact form is shown for tractable values of block size b and the approximation(under the typicality assumption) is shown for b = 64. Note that the exact and approximateforms agree for the combination b = 64, c = 2.



S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

S12

S14

S16

S11

S13

S15

sourcenodes

syndromenodes

y[1]

y[3]

y[2]

y[4]

D2

D1

symbolnodes

mappingnodes

Figure 5.5: Factor graph for multilevel decoder with parameters: number of bits per symbolm = 2, number of symbols n = 16, block size b = 4, number of candidates c = 4, Rfixed = 1

16and Radaptive = 1

2 . The associated rate-adaptive LDPC code is regular with edge-perspectivesource and syndrome degree distributions λt(ω) = ω2 and ρt(ω) = ω5, respectively, wherethe code counter t = n′

k′Radaptive.

factor graph in Fig. 4.6, there are also symbol nodes, each of which is connected

to the side information node that bears information about that symbol. Statistical

information is shared among each symbol node and its m respective source nodes

via a mapping node, which bears no information of its own. An edge incident to a

source node carries messages of the form p that represent the probability that the

source bit is 0. But an edge incident to a symbol node carries messages of the form

p = (p(0), p(1), . . . , p(2m−1)) that represent the probabilities that the source symbol


p1in

p1out

p2in

p2out

pest

(a)

pin

pout

p1in

p1out

p2in

p2out

pmin

pmout

(b)

p1in

p1out

p2in

p2out

pbin

pbout

Y1,1 Y1,2 … Y1,c

Y2,1 Y2,2 … Y2,c

Yb,1 Yb,2 … Yb,c

… … …y[i]

(c)

Figure 5.6: Multilevel factor graph nodes. Input and output messages and symbol estimate(if applicable) of (a) symbol node, (b) mapping node and (c) side information node.

is 0, 1, . . . , 2m − 1, respectively. In addition, the source and symbol nodes produce

estimates of their bit and symbol values, respectively.

The computation rules of source nodes (unattached or attached to doping nodes)

and syndrome nodes are the same as described in Section 4.3.2. We now specify the

computation rules of symbol, mapping and side information nodes, with input and

output messages and source estimates as denoted in Fig. 5.6. Keep in mind that the

sum-product algorithm requires each output message to be a function of all input

messages except the one along the same edge as that output message.

Symbol Node

Fig. 5.6(a) shows a symbol node, necessarily of degree 2. There are 2m outcomes

for the symbol random variable: it is l with likelihood weight pin1 (l)pin

2 (l), for all l ∈{0, 1, . . . , 2m−1}. Hence, the symbol estimate pest = (pest(0), pest(1), . . . , pest(2m − 1))

is given by

pest(l) =pin

1 (l)pin2 (l)

pin1 (0)pin

2 (0) + pin1 (1)pin

2 (1) + · · ·+ pin1 (2m − 1)pin

2 (2m − 1). (5.3)


To calculate the output messages pout1 and pout

2 , we ignore the input messages pin2 and

pin1 , respectively. Doing so, reduces (5.3) to

pout1 = pin

2 , (5.4)

pout2 = pin

1 . (5.5)

Mapping Node

Fig. 5.6(b) shows a mapping node, with one edge carrying multilevel probability

messages pin and pout, and m edges carrying binary probability messages pinv and pout

v

for v ∈ {1, 2, . . . ,m}. The role of the mapping node is to marginalize the output

message of the symbol or one of the bits based on all the other input symbol or bit

messages, through the mapping used in the multilevel encoder, whether binary or

Gray (or other.) We represent the mapping by the function map(l, v) which returns

the vth bit of the m bit mapping of the symbol l. Then the output probability that

the symbol is l is the product of the input probabilities of the vth bit being map(l, v)

over all v ∈ {1, 2, . . . ,m},

pout(l) =m∏v=1

(1[map(l,v)=0]p

inv + 1[map(l,v)=1](1− pin

v )). (5.6)

To compute poutu , we first calculate likelihoods wmap

u,l of the symbol being l given the

input symbol message and all the input bit messages except pinu , and then marginalize

the likelihoods over l according to whether map(l, u) = 0, as follows.

wmapu,l = pin(l)

∏v 6=u

(1[map(l,v)=0]p

inv + 1[map(l,v)=1](1− pin

v ))

(5.7)

poutu =

2m−1∑l=0

1[map(l,u)=0]wmapu,l

2m−1∑l=0

wmapu,l

(5.8)


Side Information Node

Fig. 5.6(c) shows a side information node of value y[i] consisting of b × c multilevel

symbols, all of which are labeled in Fig. 5.6(c) omitting the block index i for notational

simplicity. The derivation of an output message poutu extends the arguments in the

derivation for the side information node in Section 4.3.2 from binary to multilevel. In

the following, ? denotes circular convolution.

pnoisy-inv = pin

v ? (ε0, ε1, . . . , ε2m−1) (5.9)

wsiu,j =

∏v 6=u

2m−1∑l=0

1[Yv,j=l]pnoisy-inv (l) (5.10)

pclean-outu (l) =

c∑j=1

1[Yu,j=l]wsiu,j

c∑j=1

wsiu,j

(5.11)

poutu = pclean-out

u ? (ε0, ε1, . . . , ε2m−1) (5.12)

5.4 Analysis of Multilevel Side Information Adaptation

Using a similar process to that of Section 4.4, we determine the convergence of the

multilevel decoder for different settings of symbol-to-bit mapping and doping rate

Rfixed at the multilevel encoder. Note that our analysis requires the symmetry condi-

tions of Section 5.1 to hold.

5.4.1 Factor Graph Transformation

We transform the multilevel factor graph shown in Fig. 5.5 into a graph equivalent in

terms of convergence, shown in Fig. 5.7, using similar manipulations to the ones in

Section 4.4.1. As before, the candidates in each side information block are reordered

so that the statistically dependent one is in the first position y[i, 1]. We then subtract

from each side information candidate y[i, j] the corresponding source block x[i] mod-

ulo 2m− 1, and set all the source symbols and bits, syndrome bits and doping bits to

0. The side information nodes therefore have the first candidate y[i, 1] statistically



S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

S12

S14

S16

S11

S13

S15

sourcenodes

syndromenodes

y[1]

y[3]

y[2]

y[4]

symbolnodes

mappingnodes

Figure 5.7: Transformed multilevel factor graph equivalent to that of Fig. 5.5 in terms ofconvergence. After doping, the edge-perspective source and syndrome degree distributionsof the associated rate-adaptive LDPC code are λ∗t (ω) = ω2 and ρ∗t (ω) = 2

45ω3 + 2

9ω4 + 11

15ω5.

The edge-perspective side information and mapping degree distributions are β∗t (ω) = ω3

and µ∗t (ω) = 115 + 14

15ω.

dependent with respect to 0 and all other candidates independent of 0. Finally, we

remove the doping nodes, their attached source nodes and the edges connected to

those source nodes from the graph. To compensate for the removed edges, the map-

ping nodes must substitute the missing messages with the value 1. But, just as in

Section 4.4.1, no change is required to the syndrome node decoding rule.


5.4.2 Derivation of Degree Distributions

The multilevel factor graphs before and after transformation are each characterized

by four types of degree distribution in both edge- and node-perspective. The source

degree distributions λt(ω), Lt(ω), λ∗t (ω) and L∗t (ω) and the syndrome degree dis-

tributions ρt(ω), Rt(ω), ρ∗t (ω) and R∗t (ω) are defined and derived identically as in

Section 4.4.2. The side information degree distributions βt(ω), Bt(ω), β∗t (ω) and

B∗t (ω) have the same definitions as before, but have different derivations because the

position of multilevel side information nodes is different to their binary counterparts.

We define mapping degree distributions µt(ω), Mt(ω), µ∗t (ω) and M∗t (ω) to be the

degree distributions of the mapping nodes, counting only the mapping-source edges,

not the mapping-symbol edges.

Mapping Degree Distributions

Since the mapping nodes inhabit the same position in the multilevel factor graphs

as the side information nodes do in the binary factor graphs, their degree distribu-

tions have analogous forms. Before transformation, the edge- and node-perspective

mapping distributions are, respectively,

µt(ω) = ωm−1, (5.13)

Mt(ω) = ωm, (5.14)

because the number of mapping-source edges per mapping node is m. After trans-

formation, the node- and edge-perspective mapping degree distributions follow (4.24)

and (4.25), as

M∗t (ω) = (1− A)ωm

∗+ Aωm

∗+1, (5.15)

µ∗t (ω) =(1− A)m∗

m∗ + Aωm

∗−1 +A(m∗ + 1)

m∗ + Aωm

∗, (5.16)

where m∗ = bm(1 − Rfixed)c and A = m(1 − Rfixed) −m∗. We assume that the low

doping rate Rfixed <m−1m

, so that m∗ ≥ 1 and no mapping nodes are removed during

the factor graph transformation.


U…UE

U…UEU…UE

… … … βt*(ω)

Qsym-si

Qsi-sym

Δ2 Δ2ρt*(ω)λt*(ω)1

Qsyn-so

Qso-synQsourceΔ 111

Qmap-sym

Qsym-mapμt*(ω)

Qso-map

Qmap-so

Figure 5.8: Density evolution for multilevel decoder. The five stochastic nodes are represen-tatives of the side information, symbol, mapping, source and syndrome nodes, respectively,of a transformed multilevel factor graph, like the one in Fig. 5.7. The quantities inside thenodes are the distributions of the symbols or bits at those positions. Beside the nodes arewritten the expected edge-perspective degree distributions of the transformed multilevelfactor graph. The arrow labels are densities of messages passed in the transformed factorgraph, and Qsource is the density of source bit estimates.

Side Information Degree Distributions

The side information degree distributions are unchanged by transformation because

each side information node retains all b of its mapping node neighbors. So,

β∗t (ω) = βt(ω) = ωb−1, (5.17)

B∗t (ω) = Bt(ω) = ωb. (5.18)

5.4.3 Monte Carlo Simulation of Density Evolution

The distributions of messages passed among classes of nodes are now represented as

message densities. The densities of the binary messages to or from source nodes are

like the ones in Section 4.4.3, but the densities of multilevel messages to or from

symbol nodes are multidimensional. The densities are labeled in the schematic in

Fig. 5.8 along with five stochastic nodes representing side information, symbol, map-

ping, source and syndrome nodes. Inside the nodes are written probability distri-

butions for the symbols or bits at those positions. The symbol distributions E, U

and ∆ stand for (ε0, ε1, . . . , ε2m−1),(

12m ,

12m , . . . ,

12m

)and (1, 0, . . . , 0), respectively,

over symbol values {0, 1, . . . , 2m − 1}. The binary distribution ∆2 is (1, 0) over the

values {0, 1}. Beside the stochastic nodes are written the edge-perspective degree

distributions of the transformed factor graph.


Density evolution stochastically updates each density for a fixed number of itera-

tions. The sum-product algorithm is deemed to converge for the multilevel decoder

if and only if the source bit estimate density Qsource, after thresholding, converges to

the source bit value 0.

In our Monte Carlo simulation, we represent the densities of the binary and mul-

tilevel messages differently. A density of binary messages is a set of samples q, each

of which is a probability that its associated source bit is 0, just as in Section 4.4.3.

At initialization, all samples q are set to 12. A density of multilevel messages is a

set of multidimensional samples q = (q(0), q(1), . . . , q(2m − 1)), each of which is a

probability distribution of its associated source symbol over values {0, 1, . . . , 2m− 1}.At initialization, all samples q are set to

(1

2m ,1

2m , . . . ,1

2m

).

Several of the update rules for the densities of binary messages are similar to

those in Section 4.4.3. The rule for the syndrome-to-source message density Qsyn-so

is identical. The rules for the source bit estimate density Qsource and the source-to-

syndrome message density Qso-syn are the same as their counterparts in Section 4.4.3

with the replacement of the side-information-to-source message density Qsi-so with the

mapping-to-source message density Qmap-so. Likewise, the source-to-mapping message

density Qso-map is the same as the source-to-side-information message density Qso-si

in Section 4.4.3 with the same substitution. The update rules for the other densities

are now described.

Symbol-to-Side-Information and Symbol-to-Mapping Message Densities

Just as the symbol nodes relay messages without modification between the side in-

formation and mapping nodes in (5.4) and (5.5), the stochastic symbol node relays

densities between the side information and mapping stochastic nodes.

Qsym-si = Qmap-sym (5.19)

Qsym-map = Qsi-sym (5.20)

Side-Information-to-Symbol Message Density

To compute each updated sample qout of Qsi-sym, let a set{qinv

}b−1

v=1consist of b − 1

samples drawn randomly from Qsym-si. Create also a realization of a b × c block of


side information from the joint distribution induced by the side information stochastic

node. That is, each element Yv,j is independently drawn from E if j = 1 or from U if

j 6= 1. Generate sets{qnoisy-inv

}b−1

v=1and

{wsij

}cj=1

, before finally updating sample qout,

following (5.9) to (5.12).

qnoisy-inv = qin

v ? (ε0, ε1, . . . , ε2m−1) (5.21)

wsij =

b−1∏v=1

2m−1∑l=0

1[Yv,j=l]qnoisy-inv (l) (5.22)

qclean-out(l) =

c∑j=1

1[Yb,j=l]wsij

c∑j=1

wsij

(5.23)

qout = qclean-out ? (ε0, ε1, . . . , ε2m−1) (5.24)

Mapping-to-Symbol Message Density

To compute each updated sample qout of Qmap-sym, let a set{qinv

}mv=1

consist of δ sam-

ples drawn randomly from Qso-map and m−δ samples equal to 1. The random degree δ

is drawn equal to d with probability equal to the coefficient of ωd in node-perspective

M∗t (ω), since there is one actual output message per node. The samples set to 1

substitute for the messages on edges removed during factor graph transformation due

to doping. Then, according to (5.6),

qout(l) =m∏v=1

(1[map(l,v)=0]q

inv + 1[map(l,v)=1](1− qin

v )). (5.25)

Mapping-to-Source Message Density

To compute each updated sample qout of Qmap-so, let a set{qinv

}m−1

v=1consist of δ − 1

samples drawn randomly from Qso-map and m − δ samples equal to 1. The random

degree δ is drawn equal to d with probability equal to the coefficient of ωd−1 in edge-

perspective µ∗t (ω), since there is one actual output message per edge. Furthermore,

let qin be 1 sample drawn randomly from Qsym-map and let u be an integer drawn


randomly from {1, 2, . . . ,m}. Generate a set {wmapl }2m−1

l=0 , before updating sample

qout, following (5.7) and (5.8).

wmapl = qin(l)

∏v 6=u

(1[map(l,v)=0]q

inv + 1[map(l,v)=1](1− qin

v ))

(5.26)

qout =

2m−1∑l=0

1[map(l,u)=0]wmapl

2m−1∑l=0

wmapl

(5.27)

5.5 Multilevel Coding Experimental Results

We evaluate the coding performance of the multilevel codec under both binary and

Gray symbol-to-bit mappings. In order to compare the performance to the analysis

using density evolution, our experimental settings satisfy the symmetry conditions in

Section 5.1; in particular, each symbol comprises m = 2 bits.1 Our key finding in this

section is that both binary and Gray mappings offer coding performance close to the

Slepian-Wolf bound and are well modeled by density evolution, but that the codec

using Gray mapping usually requires a lower doping rate than the one using binary

mapping.

The source, in all our experiments, consists of n = 2048 symbols so that the

multilevel codec uses rate-adaptive LDPC codes of length n′ = mn = 4096 bits,

encoded data increment size k = 32 bits, and regular source degree distribution

λreg(ω) = ω2. Hence, Radaptive ∈{

164, 2

64, . . . , 2

}. For convenience, we allow Rfixed ∈{

0, 164, 2

64, . . . , 1

4

}. The statistical dependence between source blocks and their match-

ing side information candidates is parameterized by σ as in (5.1) and (5.2). The

Monte Carlo simulation of density evolution uses up to 214 samples.

Fig. 5.9 to Fig. 5.12 fix the values of block size and number of candidates (b, c)

to the combinations (8, 2), (32, 2), (64, 4) and (64, 16), respectively. In each figure,

for both binary and Gray mappings, we determine the optimal doping rates Rfixed at

different values of H(σ), and plot the multilevel codec’s empirical performance and

performance modeled by density evolution. The corresponding Slepian-Wolf bounds

1For experiments using larger values of m, refer to the results in Chapter 6.


are computed exactly in Fig. 5.9 and Fig. 5.10 and approximately in Fig. 5.11 and

Fig. 5.12 using the derivations in Section 5.2.2.

These figures demonstrate that, for m = 2 and regardless of the choice of mapping

and the settings of b and c, the performance of the multilevel codec with optimal

doping is close to the Slepian-Wolf bound and is modeled well using density evolution.

Whether the binary or Gray mapping provides better compression depends on the

values of b, c and H(σ), but the Gray mapping usually uses a lower optimal doping

rate.

5.6 Summary

This chapter extends the algorithms and analysis of side-information-adaptive coding

from binary to multilevel source and side information. There are two new chal-

lenges: the need for whole symbol coding and the requirement that certain symmetry

conditions hold for density evolution to work. We also extend the block-candidate

statistical model to the multilevel case and derive the Slepian-Wolf bound. The whole

symbol encoder converts the multilevel source into bits using either a binary or Gray

mapping and then applies the binary side-information-adaptive encoder. The whole

symbol decoder recovers the multilevel symbols using the factor graph of the binary

side-information-adaptive decoder augmented with symbol and mapping nodes. We

analyze the multilevel extension by transforming the augmented factor graph into

one equivalent in terms of convergence, deriving its degree distributions under vary-

ing rates of doping, and applying a Monte Carlo simulation of density evolution. Our

experimental results under a variety of settings of the block-candidate model show

that both binary and Gray mappings offer coding performance close to the Slepian-

Wolf bound and well modeled by density evolution analysis, but that Gray mapping

usually requires a lower doping rate than binary mapping.


0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)

Performance with Binary m=2, b=8, c=2

Multilevel codecDE modelSW boundH(σ)Doping rate

(a)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)ra

te (

bit/s

ymbo

l)

Performance with Gray m=2, b=8, c=2


(b)

Figure 5.9: Coding performance of multilevel codec with optimal doping rate Rfixed, m = 2,b = 8, c = 2 and either (a) binary or (b) Gray mapping.

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)



(a)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)



(b)



0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)



(a)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)ra

te (

bit/s

ymbo

l)



(b)


0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)



(a)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)



(b)


Chapter 6

Applications

This chapter applies the distributed source coding algorithms developed in this dis-

sertation to reduced-reference video quality monitoring, multiview coding and low-

complexity video encoding. These topics were introduced and their literature was

surveyed in Section 2.3. All three applications make use of rate adaptation and side

information adaptation with multilevel signals, but our experiments for each applica-

tion showcase different aspects of performance.

In Section 6.1 on reduced-reference video quality monitoring, we apply density

evolution analysis to design distributed source coding algorithms for the coding of

video projection coefficients. Section 6.2 on multiview coding demonstrates that

distributed source coding outperforms state-of-the-art multiview image coding based

on separate encoding and decoding. In Section 6.3 on low-complexity video coding,

we show that distributed video coding with side information adaptation for motion

outperforms coding without side information adaptation when there is motion.

6.1 Reduced-Reference Video Quality Monitoring

End-to-end quality monitoring is an important service in the delivery of video over

Internet Protocol (IP) networks. Fig. 6.1 depicts a video transmission system with an

attached system for reduced-reference video quality monitoring and channel tracking.

A server sends c channels of encoded video to an intermediary. The intermediary

transcodes a single trajectory with channel changes through the video and forwards

77

CHAPTER 6. APPLICATIONS 78

VideoEncoder

VideoDecoder

c videochannels

1 videotrajectory

VideoTranscoder

J.240Projection

J.240Projection

ProjectionDecoder

ProjectionEncoder

PSNREstimator

Y X

Server Intermediary Device

rate controlfeedback

encodedcoefficients

Figure 6.1: Video transmission with quality monitoring and channel tracking.

it to a display device.1 We are interested in the attached monitoring and tracking

system. The device encodes projection coefficients X of the video trajectory and

sends them back to the server. A feedback channel from server to device is available

for rate control. The server decodes and compares the coefficients with projection

coefficients Y of the c channels of video, in order to estimate the transcoded video

quality and track its channel change.

This section first describes the projection and peak signal-to-noise ratio (PSNR)

estimation technique of the ITU-T J.240 standard [2] for reduced-reference video

quality monitoring. We suggest an improved maximum likelihood PSNR estimation

technique, applicable when the PSNR estimator is located at the server. The main

contribution of this section is the design by density evolution analysis of distributed

source coding systems for the projection coefficients. These coding systems signifi-

cantly reduce the bit rate needed for quality monitoring and channel tracking.

1An example of such an intermediary is the Slingbox made by Sling Media Inc.


2D WHT 2D IWHT Downsampling

MaximumLength

Sequence 1

MaximumLength

Sequence 2

Figure 6.2: Dimension reduction projection of ITU-T J.240 standard.

6.1.1 ITU-T J.240 Standard

The J.240 standard for reduced-reference video quality monitoring specifies both a

dimension reduction projection for video signals and a function that compares two

sets of coefficients to estimate PSNR [2].

The projection partitions the luminance channel of the video into blocks, sizes

of 8 × 8 or 16 × 16 pixels being typical. From each block, a single coefficient is

obtained by the process shown in Fig. 6.2. The block is multiplied by a maximum

length sequence [76], transformed using the 2D Walsh-Hadamard transform (WHT),

multiplied by another maximum length sequence, and inverse transformed using the

2D inverse Walsh-Hadamard transform (IWHT). Finally, one coefficient is sampled

from the block.

Suppose that χ and ϕ, each of length n, denote projection coefficient vectors of a

transcoded trajectory and its matching encoded counterpart, respectively. The J.240

standard assumes that both vectors are uniformly quantized with step size Q into χ

and ϕ. The PSNR of the transcoded video with respect to the original encoded video

is estimated as

MSEJ.240 =Q2

n

n∑i=1

(χi − ϕi)2 (6.1)

PSNRJ.240 = 10 log10

2552

MSEJ.240

. (6.2)


6.1.2 Maximum Likelihood PSNR Estimation

When the PSNR estimator is located at the server (as in Fig. 6.1), it has access to

the unquantized coefficient vector Y. We suggest the following maximum likelihood

(ML) estimation formulas, which support nonuniform quantization of X, in [113].

MSEML = E

[1

n

n∑i=1

(χi − ϕi)2

∣∣∣∣χi, ϕi]

(6.3)

PSNRML = 10 log10

2552

MSEML

(6.4)

We now compare the performance of J.240 and maximum likelihood PSNR es-

timation for c = 8 channels of video (Foreman, Carphone, Mobile, Mother and

Daughter, Table, News, Coastguard, Container) at resolution 176×144 and frame

rate 30 Hz. The first 256 frames of each sequence are encoded at the server using

H.264/AVC Baseline profile with quantization parameter set to 16 [217]. The interme-

diary transcodes each trajectory with JPEG using scaled versions of the quantization

matrix specified in Annex K of the standard [1], with scaling factors 0.5, 1 and 2,

respectively. Fig. 6.3 shows encoded and transcoded versions of frame 0 of the Fore-

man sequence. The difference in quality of the transcoded images is noticeable both

visually and through their PSNR values of 36.9, 35.2 and 33.6 dB, respectively.

The number of J.240 coefficients per frame, denoted b, is either 99 or 396, depend-

ing on the block size 16× 16 or 8× 8, respectively. The coefficients are quantized to

m bits, uniformly for estimation by the J.240 standard and nonuniformly for max-

imum likelihood estimation. The nonuniform quantization is designed to populate

the quantization bins approximately equally with samples from the entire data set.

Fig. 6.4(a) and (b) plot the mean absolute PSNR estimation error versus the PSNR

estimation group of picture (GOP) size for b = 99 and 396, respectively. The PSNR

estimation GOP size is the number of frames over which the estimation is performed.

Maximum likelihood estimation of PSNR at just m = 1 or 2 bits outperforms J.240

estimation at m = 7 bits and, in some cases, even m = 9 bits. Observe also that max-

imum likelihood estimation improves significantly as the number of coefficients in the

PSNR estimation GOP grows. This suggests that video quality monitoring for video

at low resolution (such as 176×144) is more challenging than at higher resolution.


(a) (b)

(c) (d)

Figure 6.3: Frame 0 of Foreman sequence at resolution 176×144, (a) encoded usingH.264/AVC, and (b)–(d) transcoded using JPEG with quantization matrix scaling factors0.5, 1 and 2, respectively, to PSNR of 36.9, 35.2 and 33.6 dB.

6.1.3 Distributed Source Coding of J.240 Coefficients

The J.240 standard takes for granted that the projection encoder in Fig. 6.1 uses sim-

ple fixed length coding. Conventional variable length coding produces limited gains

because the coefficients X are approximately independent and identically distributed

Gaussian random variables, due to the dimension reduction projection in Fig. 6.2. In

contrast, distributed source coding of X offers better compression by exploiting the


1 2 4 8 16 32 64 128 2560

1

2

3

4

PSNR estimation GOP size

mea

n ab

solu

te P

SN

R e

rror

(dB

)PSNR Estimation Error with b=99

ML estimation with m=1ML estimation with m=2J.240 standard with m=7J.240 standard with m=8J.240 standard with m=9

(a)

1 2 4 8 16 32 64 128 2560

1

2

3

4

PSNR estimation GOP sizem

ean

abso

lute

PS

NR

err

or (

dB)

PSNR Estimation Error with b=396

ML estimation with m=1ML estimation with m=2J.240 standard with m=8J.240 standard with m=9J.240 standard with m=10

(b)

Figure 6.4: PSNR estimation error using the J.240 standard and maximum likelihood (ML)estimation with m bit quantization for number of coefficients per frame (a) b = 99 and (b)b = 396.

statistically related coefficients Y at the projection decoder. This side information

adheres to the block-candidate model with block size equal to b, the number of J.240

coefficients per frame, and number of candidates equal to c, the number of chan-

nels. In the remainder of this section, we design and evaluate six adaptive distributed

source coding systems for this task.

Table 6.1 presents settings for these codecs. The independent settings are m = 1 or

2 (with binary or Gray symbol-to-bit mapping for m = 2), b = 99 or 396 and c = 8.

For each codec, the decoder assumes a value either ε or σ (depending on whether

m = 1 or 2) that parameterizes the distribution between the J.240 coefficients of

a transcoded frame and those of the respective encoded frame at the server. We

choose ε and σ giving entropies H(ε) = 0.2 and H(σ) = 0.4 bit/coefficient, because

these entropies exceed about 90% of the empirical conditional entropy rates. We

also design the doping rates Rfixed by applying the analysis algorithms developed in

Sections 4.4 and 5.4 as follows. The optimal coding and doping rates, as predicted by

density evolution under the independent settings, are plotted in Fig. 6.5 for the entire


Independent Settings Dependent Settings

m Mapping b c Decoder ε Decoder σ Rfixed

1 99 8 0.0311 0.03031 396 8 0.0311 0.01522 binary 99 8 0.3835 0.04552 Gray 99 8 0.3835 0.03032 binary 396 8 0.3835 0.01522 Gray 396 8 0.3835 0.0152

Table 6.1: Codec settings for distributed source coding of J.240 coefficients.

ranges 0 ≤ H(ε) ≤ 1 or 0 ≤ H(σ) ≤ 2. The optimal doping rates are chosen from{0, m

132, 2m

132, . . . , 16m

132

}. For each codec, we choose the value of Rfixed as the maximum

optimal doping rate for H(ε) ≤ 0.2 or H(σ) ≤ 0.4. In these ranges, observe that the

coding rates are lower for binary mapping than Gray mapping, when m = 2.

All six systems are tested using the same video sequences, coding, transcoding

and quantization that generate the maximum likelihood PSNR estimation curves

with m = 1 and 2 and b = 99 and 396 in Fig. 6.4. Although the codecs are designed

for c = 8 channels of video, we evaluate their performance with c = 1, 2, 4 and 8.

In each trial, the coefficients X are obtained from a transcoded random trajectory

through c of the channels. The coefficients Y in block-candidate form are obtained

from the versions of the same c channels, encoded at the server. The codecs process 8

frames of coefficients at a time, using rate-adaptive LDPC codes with regular source

degree distribution nreg(ω) = ω2, length n′ = nm = 8bm bits and data increment

size k = 8bm132

bits. Consequently, Radaptive ∈{m

132, 2m

132, . . . ,m

}. Fig. 6.6 compares the

average coding rates in bit/coefficient for the six systems when c = 1, 2, 4 and 8.

Even though the codecs were designed for c = 8, they perform better or as well when

c is less than 8. The coding rates for c = 8 agree with those predicted by density

evolution. In particular, coding is more efficient for m = 2 bit binary mapping than

m = 2 bit Gray mapping.

Fig. 6.7 plots the mean absolute PSNR estimation error versus the estimation bit

rate for different combinations of estimation and coding techniques, all with c = 8.

The PSNR estimates are computed over a PSNR estimation GOP size of 256 frames.

The estimation bit rate is the transmission rate from projection encoder to projection


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)

rate

(bi

t/bit)

Density Evolution with m=1, b=99, c=8

DE modelSW boundH(ε)Doping rate

(a)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

H(ε) (bit/bit)ra

te (

bit/b

it)


DE modelSW boundH(ε)Doping rate

(b)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)


Binary DE modelGray DE modelSW boundH(σ)Binary doping rateGray doping rate

(c)

0 0.4 0.8 1.2 1.6 20

0.4

0.8

1.2

1.6

2

H(σ) (bit/symbol)

rate

(bi

t/sym

bol)


Binary DE modelGray DE modelSW boundH(σ)Binary doping rateGray doping rate

(d)

Figure 6.5: Analysis of distributed source coding of J.240 coefficients using density evolu-tion (DE) with optimal doping rate Rfixed, under the settings (a) m = 1, b = 99, c = 8,(b) m = 1, b = 396, c = 8, (c) m = 2, b = 99, c = 8 with binary and Gray mapping and(d) m = 2, b = 396, c = 8 with binary and Gray mapping.


m=1 binary m=2 Gray m=2 0

0.1

0.2

0.3

0.4

0.5

0.6

codi

ng r

ate

(bit/

coef

ficie

nt)

Distributed Source Coding with b=99

c=1c=2c=4c=8

(a)

m=1 binary m=2 Gray m=2 0

0.1

0.2

0.3

0.4

0.5

0.6

codi

ng r

ate

(bit/

coef

ficie

nt)

Distributed Source Coding with b=396

c=1c=2c=4c=8

(b)

Figure 6.6: Coding rate for distributed source coding of J.240 coefficients, with settings asin Table 6.1, including number of J.240 coefficients per frame b set to (a) 99 and (b) 396.

decoder in kbit/s assuming the video channels have frame rate of 30 Hz. In all trials,

the transcoded trajectory is correctly tracked and the PSNR estimation errors are

computed with respect to the matching trajectory at the server. The curve for J.240

estimation and fixed length coding is obtained by varying the number of quantization

bits m from 1 to 10. The performance of maximum likelihood estimation and fixed

length coding is shown only for m = 1 and 2. The combination of maximum likelihood

estimation and distributed source coding is also evaluated for m = 1 and 2, the

latter with binary symbol-to-bit mapping. The best combination that achieves PSNR

estimation error around 0.5 dB requires an estimation rate of only 1.27 kbit/s; it uses

maximum likelihood estimation and distributed source coding with b = 99, m = 2

and binary mapping. This performance is almost 20 times better than that of J.240

estimation and fixed length coding, which requires an estimation rate of 23.7 kbit/s

for about 0.5 dB PSNR estimation error.


0 5 10 15 20 25 300

5

10

15

20

25

30

estimation rate (kbit/s)

mea

n ab

solu

te P

SN

R e

rror

(dB

)Performance with b=99, c=8

J.240 estimation and FLCML estimation and FLCML estimation and DSC

(a)

0 20 40 60 80 100 1200

5

10

15

20

25

30

estimation rate (kbit/s)m

ean

abso

lute

PS

NR

err

or (

dB)

Performance with b=396, c=8

J.240 estimation and FLCML estimation and FLCML estimation and DSC

(b)

Figure 6.7: Mean absolute PSNR estimation error versus estimation bit rate for differentestimation and coding techniques, with number of J.240 coefficients per frame b set to (a) 99and (b) 396. PSNR estimation is either by J.240 or maximum likelihood (ML) and codingis either fixed length coding (FLC) or distributed source coding (DSC).

6.2 Multiview Coding

Arrays of tens or hundreds of cameras capture correspondingly large amounts of raw

data. Distributed source coding of these views at separate encoders, one for each

camera, reduces the raw data without bringing all of it to a single encoder. The

separately encoded bit streams are sent to a joint decoder, which reconstructs the

views. Section 4.1 introduces a toy version of this problem, the adaptive distributed

source coding of random dot stereograms [91]. We now extend that idea to the

practical lossy transform-domain coding of real multiview images from arrays of more

than one hundred cameras. We demonstrate that distributed multiview image coding

outperforms codecs based on separate encoding and decoding.

Our system requires that one key view be conventionally coded and reconstructed.

Each of the other views is coded as depicted in Fig. 6.8, where X and X are the view

and its reconstruction and Y is another already reconstructed view. At the encoder,

the view X is first transformed using an 8 × 8 discrete cosine transform (DCT) and


LDPCEncoder

rate controlfeedback

Side InformationAdapter

Transform Quantizer InverseTransform

Recon-struction

OvercompleteTransform

LDPCDecoder

Figure 6.8: Lossy transform-domain adaptive distributed source codec. The core is anadaptive distributed source codec that performs lossless coding of quantization indices.

quantized using the matrix in Annex K of the JPEG standard, scaled by some fac-

tor [1]. The resulting m-bit quantization indices X are compressed by the LDPC

encoder, which is part of an adaptive distributed source codec. At the decoder, the

overcomplete transform computes the DCT of all 8 × 8 blocks of the reconstructed

view Y and supplies the side information adapter with block-candidate side informa-

tion Y. The side information consists of c candidates for each block of b indices in X,

derived from an 8× 8 block of X. The LDPC decoder and side information adapter

iteratively decode the quantization indices, requesting increments of rate from the

LDPC encoder using the feedback channel as necessary. The quantization coeffi-

cients are reconstructed to the centroids of their quantization intervals2 and inverse

transformed, producing the reconstructed view X.

We apply this coding technique to two multiview image data sets Xmas and

Dog with camera geometry as described in Table 6.2 [195, 196]. Each view is a

luminance signal of resolution 320 × 240. View 0 is set as a key view and coded

conventionally with JPEG. For each of the other views, view i is coded as X according

to Fig. 6.8 using the reconstructed view i−1 as Y . The views are quantized with one

of four scaling factors 0.5, 0.76, 1.15 and 1.74, which make m = 8 bits sufficient to

2Although side-information-assisted reconstruction is useful when alternating views are key views [11], itdegrades performance when there is just one key view.


Data Camera Camera Number of Inter-Camera Distance toSet Array Alignment Cameras Distance (m) Scene (m)

Xmas linear horizontal parallel 101 0.003 0.3Dog linear horizontal converging 80 0.05 8.2

Table 6.2: Camera array geometry for multiview data sets Xmas and Dog [195, 196].

represent the indices. We choose a Gray mapping from symbols to bits to minimize

the number of bit transitions between adjacent symbols. The side-information’s block

size b is fixed at 64 coefficients because of the 8 × 8 transforms at encoder and

decoder. The linear horizontal camera arrangement means that the 8× 8 candidates

in Y for each 8 × 8 block of X lie in a horizontal search range. For these data sets,

searching through integer shifts in [−5, 5] is enough, giving c = 11 candidates per

block. We statistically model the difference between a source block and its matching

side information candidate as zero-mean Laplacian distributions, one tuned for each

of the b = 64 coefficients. In our experiments, we divide each view into 10 horizontal

tiles of size 320× 24 pixels and code each tile individually. The rate-adaptive LDPC

codes therefore have length n = 61440 bits with data increment size k = 480 bits and

regular source degree distribution nreg(ω) = ω2. To determine the doping rate Rfixed,

we do not perform analysis by density evolution since the source and side information

do not satisfy all the required assumptions. For example, each block of the source is

usually statistically dependent on all the candidates (instead of exactly one) in the

side information block, especially in the low frequency coefficients. Instead, we find

that Rfixed = 0 suffices, precisely due to the redundant side information.

Fig. 6.9 shows that the overall rate-distortion performance of the adaptive codec

is superior to separate encoding and decoding of the views. In coding Xmas and

Dog, it outperforms the intra mode of H.264/AVC Baseline by about 3 dB and 0.5

dB, respectively, and JPEG by about 6 dB and 4 dB, respectively. We also compare

the performance of an oracle codec, which operates in almost the same way as the

adaptive codec. The difference is that the side information adapter in Fig. 6.8 is

replaced with an oracle that knows exactly which candidates of Y match the blocks

of X. That the adaptive codec performs almost as well as the oracle codec supports

our choice of Rfixed = 0.


0.2 0.4 0.6 0.8 1 1.2 1.429

31

33

35

37

39

rate (bit/pixel)

PS

NR

(dB

)RD Performance, Xmas

OracleSide Information AdaptationH.264/AVC IntraJPEG

(a)

0.3 0.5 0.7 0.9 1.1 1.3 1.529

31

33

35

37

rate (bit/pixel)P

SN

R (

dB)

RD Performance, Dog

OracleSide Information AdaptationH.264/AVC IntraJPEG

(b)

Figure 6.9: Comparison of rate-distortion (RD) performance for multiview coding of theoracle, adaptive, H.264/AVC intra and JPEG codecs, for the data sets (a) Xmas and(b) Dog.

Xmas Dog

Codec PSNR (dB) rate (bit/pixel) PSNR (dB) rate (bit/pixel)

Adaptive 37.9 0.545 36.7 0.828H.264/AVC intra 35.1 0.549 36.1 0.838

JPEG 31.5 0.542 32.9 0.834

Table 6.3: Average PSNR and rate for multiview coding traces in Fig. 6.10.

In Fig. 6.10, we plot PSNR and rate traces for the adaptive, H.264/AVC intra

and JPEG codecs, when they operate at approximately the same rates, which are

shown together with average PSNR in Table 6.3. The PSNR traces in Fig. 6.10(a)

and (b) indicate that the adaptive codec consistently reconstructs views better than

H.264/AVC intra and JPEG. The Dog data set has greater PSNR variability across

views than Xmas because its converging camera geometry makes its images less uni-

form. In the rate traces in Fig. 6.10(c) and (d), the fact that the adaptive codec

codes view 0 conventionally creates rate peaks at view 0. Even besides that view, the


0 20 40 60 80 10031

32

33

34

35

36

37

38

39

40

view number

PS

NR

(dB

)PSNR Trace, Xmas

Side Information AdaptationH.264/AVC IntraJPEG

(a)

0 20 40 60 8032

33

34

35

36

37

38

39

view numberP

SN

R (

dB)

PSNR Trace, Dog


(b)

0 20 40 60 80 1000.5

0.6

0.7

0.8

0.9

1

1.1

1.2

view number

rate

(bi

t/pix

el)

Rate Trace, Xmas


(c)

0 20 40 60 800.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

view number

rate

(bi

t/pix

el)

Rate Trace, Dog


(d)

Figure 6.10: PSNR and rate traces for multiview coding. The adaptive, H.264/AVC intraand JPEG codecs operate at similar rates, which are shown together with average PSNR inTable 6.3. The traces are for (a) PSNR of Xmas, (b) PSNR of Dog, (c) rate of Xmas and(d) rate of Dog.


adaptive codec has greater rate variability than the other two codecs because its rate

depends on the statistics of pairs of views.

From the traces in Fig. 6.10, we select pairs of images, views 35 and 65 from Xmas

and views 35 and 45 from Dog, for stereographic viewing in Fig. 6.11. The adaptive

codec’s reconstructions in Fig. 6.11(a) and (b) retain more detail than those of the

H.264/AVC intra codec in Fig. 6.11(c) and (d), for example in the reins of the sleigh in

Xmas and the creases of the curtain in Dog. They also have fewer block compression

artifacts compared to the JPEG reconstructions in Fig. 6.11(e) and (f).

6.3 Low-Complexity Video Encoding

Motion-compensated hybrid coding of video relies on a computationally intensive

encoder to exploit the redundancy among frames of video. Distributed source coding

provides a way for the frames to be encoded separately, and thus at lower complexity,

while having their joint redundancy exploited at the decoder. In this section, we

describe a distributed video codec based on adaptive distributed source coding and

similar to the multiview image codec in Section 6.2. The main result is that the

adaptive codec (with side information adaptation) outperforms the nonadaptive codec

(that does not adapt to motion), when there is motion in the video. When there is

little motion, the adaptive codec performs just as well as the nonadaptive one.

Our experiments use standard video test sequences Foreman, Carphone, Container

and Hall (listed in order of decreasing motion activity) at resolution 352 × 288 and

frame rate 30 Hz. The raw video is in YUV 4:2:0 format, which means that the

luminance channel Y has resolution 352×288 and the two chrominance channels U and

V have resolution 176×144. In keeping with video coding practice, every eighth frame

starting from frame 0 is intra coded as a key frame, in our case, using JPEG. For the

remaining frames, the luminance channel of frame i is coded as X according to Fig. 6.8

with the luminance channel of the reconstructed frame i− 1 as Y . The chrominance

channels are coded without side information adaptation. Instead, the motion vectors

obtained from the luminance decoding are used to generate motion-compensated side

information for the chrominance channels. All channels are transformed using 8× 8

DCT and quantized using a scaled version of the quantization matrix in Annex K


(a) (b)

(c) (d)

(e) (f)

Figure 6.11: Stereographic reconstructions of pairs of multiview images, views 35 and 65 ofXmas and views 35 and 45 of Dog from the traces in Fig. 6.10, through coding by (a) and(b) adaptive, (c) and (d) H.264/AVC intra, and (e) and (f) JPEG codecs.


of the JPEG standard with one of four scaling factors 0.5, 1, 2 and 4 [1]. Like

the multiview image codec, quantization is to m = 8 bits with Gray symbol-to-bit

mapping and the side-information’s block size b = 64 coefficients. The candidates in Y

for each 8×8 block of X lie in a motion search range of [−5, 5]×[−5, 5], giving c = 121

candidates per block. Also like the multiview image codec, we model the difference

between a source block and its matching side information candidate as tuned zero-

mean Laplacian distributions for each coefficient. For the experiments, the luminance

and two chrominance channels are divided into 16, 4 and 4 tiles, respectively, each

of size 88 × 72 pixels, and each tile is coded individually. The rate-adaptive LDPC

codes have length n = 50688 bits with data increment size k = 384 bits and regular

source degree distribution nreg(ω) = ω2. We do not perform density evolution analysis

to determine the doping rate Rfixed for the same reasons as for the multiview image

codec. As before, we find that Rfixed = 0 is enough.

Fig. 6.12 compares the overall rate-distortion performance of four codecs in coding

the first 16 frames of each sequence. All the codecs have similar encoding complexity

because they use the same sets of transforms and quantization. As in Section 6.2,

the oracle codec is the same as the adaptive codec, except that the side information

adapter in Fig. 6.8 is replaced with an oracle that knows exactly which candidates of

Y match the blocks of X. The nonadaptive codec is the same as the adaptive codec,

except that the side information adapter is replaced with a module that always selects

the zero motion candidate.

The adaptive codec performs almost as well as the oracle codec, for all four video

sequences, again validating our choice of Rfixed = 0. For Foreman, the adaptive codec

clearly outperforms its nonadaptive counterpart due to the significant motion in the

sequence. With less motion in Carphone, the gap is reduced. When there is almost

no motion as in Container and Hall, the nonadaptive codec functions equivalently to

the oracle codec, and so is marginally superior to the adaptive codec.

We plot luminance PSNR and rate traces in Fig. 6.13 and 6.14, respectively. The

traces are selected so that the four codecs plotted in the same panel operate at the

same PSNR. The average PSNR and rates for these traces are shown in Table 6.4. At

frames 0 and 8, all the codecs perform identically since every eighth frame is always

coded by JPEG. These traces agree with the rate-distortion plots in Fig. 6.12.


0 1 2 3 4 5 6 727

29

31

33

35

37

rate (Mbit/s)

lum

inan

ce P

SN

R (

dB)

RD Performance, Foreman

OracleSide Information AdaptationNo Side Information AdaptationJPEG

(a)

0 1 2 3 4 5 6 729

31

33

35

37

39

rate (Mbit/s)lu

min

ance

PS

NR

(dB

)

RD Performance, Carphone


(b)

0 1 2 3 4 5 6 728

30

32

34

36

38

rate (Mbit/s)

lum

inan

ce P

SN

R (

dB)

RD Performance, Container


(c)

0 1 2 3 4 5 6 730

32

34

36

38

40

rate (Mbit/s)

lum

inan

ce P

SN

R (

dB)

RD Performance, Hall


(d)

Figure 6.12: Comparison of rate-distortion (RD) performance for distributed video codingof the oracle, adaptive, JPEG and nonadaptive codecs, for the sequences (a) Foreman,(b) Carphone, (c) Container and (d) Hall in YUV 4:2:0 format at resolution 352× 288 andframe rate 30 Hz.


Foreman Carphone

Codec PSNR (dB) rate (Mbit/s) PSNR (dB) rate (Mbit/s)

Oracle 34.4 2.48 35.4 2.04Adaptive 34.4 2.99 35.4 2.20

Nonadaptive 34.4 4.51 35.4 2.81JPEG 34.4 2.92 35.4 2.53

Container Hall

Codec PSNR (dB) rate (Mbit/s) PSNR (dB) rate (Mbit/s)

Oracle 34.0 1.53 36.6 1.80Adaptive 34.0 1.60 36.6 1.85

Nonadaptive 34.0 1.53 36.6 1.84JPEG 34.0 3.03 36.6 2.68

Table 6.4: Average luminance PSNR and rate for distributed video coding traces in Fig. 6.13and 6.14.

Fig. 6.15 depicts the reconstructions of frame 15 of the sequences Foreman, Car-

phone, Container and Hall as coded by the adaptive codec in the PSNR and rate

traces in Fig. 6.13 and 6.14. These reconstructions show that the PSNR measure-

ments are matched by good visual quality.

6.4 Summary

This chapter demonstrates the potential of adaptive distributed source coding for

three applications, while highlighting different facets of its performance. We design

distributed source coding algorithms using density evolution analysis for the prob-

lem of reduced-reference video quality monitoring and channel tracking. We show

that distributed multiview image coding outperforms state-of-the-art coding based

on separate encoding and decoding. Finally, for low-complexity video encoding of se-

quences with motion, we demonstrate that coding with side information adaptation

for motion delivers better performance than coding that does not adapt to motion.


0 3 6 9 12 1533

34

35

36

frame number

lum

inan

ce P

SN

R (

dB)

PSNR Trace, Foreman


(a)

0 3 6 9 12 1534

35

36

37

frame numberlu

min

ance

PS

NR

(dB

)

PSNR Trace, Carphone


(b)

0 3 6 9 12 1533

34

35

36

frame number

lum

inan

ce P

SN

R (

dB)

PSNR Trace, Container


(c)

0 3 6 9 12 1535

36

37

38

frame number

lum

inan

ce P

SN

R (

dB)

PSNR Trace, Hall


(d)

Figure 6.13: Luminance PSNR traces for distributed video coding. The oracle, adaptive,JPEG and nonadaptive codecs operate at identical PSNR, shown together with averagerates in Table 6.4. The PSNR traces are for (a) Foreman, (b) Carphone, (c) Container and(d) Hall, and correspond to the respective rate traces in Fig. 6.14.


0 3 6 9 12 151

2

3

4

5

6

7

8

frame number

rate

(M

bit/s

)Rate Trace, Foreman


(a)

0 3 6 9 12 151

2

3

4

5

frame numberra

te (

Mbi

t/s)

Rate Trace, Carphone


(b)

0 3 6 9 12 151

2

3

4

frame number

rate

(M

bit/s

)

Rate Trace, Container


(c)

0 3 6 9 12 151

2

3

4

frame number

rate

(M

bit/s

)

Rate Trace, Hall


(d)

Figure 6.14: Rate traces for distributed video coding. The oracle, adaptive, JPEG andnonadaptive codecs operate at similar PSNR, which are shown together with average ratesin Table 6.4. The rate traces are for (a) Foreman, (b) Carphone, (c) Container and (d) Hall,and correspond to the respective PSNR traces in Fig. 6.13.


(a) (b)

(c) (d)

Figure 6.15: Reconstructions of distributed video coding frames, frame 15 from the traces inFig. 6.13 and 6.14, of the sequences (a) Foreman, (b) Carphone, (c) Container and (d) Hall.

Chapter 7

Conclusions

Adaptive distributed source coding is the separate encoding and joint decoding of

statistically dependent signals, when there is uncertainty in the statistical dependence.

This dissertation considers the encoder and decoder to have separate access to the

source and side information, respectively, and addresses adaptive distributed source

coding with three intertwining threads of study: algorithms, analysis and applications.

Algorithms

We develop coding algorithms for rate adaptation and side information adaptation

for both binary and multilevel signals.

Rate adaptation is the existing idea that the encoder switches flexibly among

coding rates. This capability, combined with a feedback channel from the decoder,

means that the encoder need not know in advance the degree of statistical depen-

dence between source and side information. Our contribution is a construction for

rate-adaptive low-density parity-check (LDPC) codes that uses syndrome merging

and splitting rather than naıve syndrome deletion. These codes perform close to

the information-theoretic bounds and are able to outperform commonly used rate-

adaptive turbo codes. Furthermore, the LDPC code construction facilitates decoding

by the sum-product algorithm and so permits performance analysis by density evo-

lution, which we discuss in more detail below.

Side information adaptation is our novel technique for the decoder to adapt to

multiple candidates of side information without knowing in advance which candidate

99

CHAPTER 7. CONCLUSIONS 100

is most statistically dependent on the source. Our approach defines block-candidate

models for side information, for which we compute tight information-theoretic coding

bounds and devise decoders based on the sum-product algorithm. The main experi-

mental finding is that the encoder usually sends a low rate of the source bits uncoded

as doping bits in order to achieve coding performance close to the bounds. In the case

of multilevel signals, we argue that side information adaptation requires the coding

of whole symbols rather than coding bit-plane-by-bit-plane. We also observe that

binary and Gray symbol-to-bit mappings yield different coding performance.

Future work in adaptive distributed source coding algorithms should continue

adding functionality. An unsolved problem is the construction of layered rate-adaptive

codes, for which some specified bits of the source are recovered at rates insufficient for

complete decoding. Side information adaptation might be extended to more general

block-candidate models, for example, ones symmetric in source and side information.

Analysis

The analysis of coding performance employs density evolution, a technique we ap-

ply to all the decoding algorithms in this dissertation. The idea is that testing the

convergence of distributions (or densities) of sum-product messages is more efficient

than testing the convergence of the messages themselves, because the former does not

require complete knowledge of the decoder’s factor graph.

Experiments demonstrate that the analysis technique closely approximates em-

pirical coding performance, and consequently enables tuning of parameters of the

coding algorithms. In this way, we design the aforementioned rate-adaptive LDPC

codes that outperform rate-adaptive turbo codes and select the optimal doping rates

for side information adaptation.

Our analysis for multilevel signals requires that certain symmetry conditions hold.

In particular, symbol values must map to bit representations in such a way that,

whenever the bits of any subset of bit planes are flipped, the new circular ordering is

isomorphic to the original ordering. We show that this isomorphism condition holds

for 2-bit symbols with either binary or Gray mapping. Extending density evolution

analysis to symbols of greater than 2 bits remains an open problem.

CHAPTER 7. CONCLUSIONS 101

Applications

We showcase this dissertation’s algorithm and analysis contributions by demonstrat-

ing their use in different media applications.

End-to-end quality is a key concern in the delivery of video over best-effort net-

works. We devise a reduced-reference video quality monitoring and channel tracking

system based on adaptive distributed source coding, and design it using density evo-

lution analysis. The reduced-reference bit rate is almost 20 times lower than that of

a comparable system based on the ITU-T J.240 standard.

We develop a lossy multiview coding system that adapts to uncertain statistics

caused by unknown disparity among the views. Operating on image data of 101

views, our codec achieves a quality 3 dB higher in peak signal-to-noise ratio (PSNR)

than that of intra coding with H.264/AVC Baseline at the same bit rate.

Finally, we apply our adaptive techniques to a classic distributed source coding

problem: low-complexity video encoding. Our codec adapts to the motion in the

video and, whenever there is motion, outperforms a codec that assumes zero motion.

These media systems demonstrate the potential of adaptive distributed source

coding using readily accessible image and video data. In contrast, as suggested in

the introduction to this dissertation, the most exciting applications, such as satellite

imaging and capsule endoscopy, involve hard-to-obtain data. It is just this inacces-

sibility that fits them to the constraint of separate encoding and joint decoding and

that continues to leave them open to future systems research.

Bibliography

[1] ITU-T Recommendation T.81: Digital compression and coding of continuous-

tone still images. Sept. 1992.

[2] ITU-T Recommendation J.240: Framework for remote monitoring of transmit-

ted picture signal-to-noise ratio using spread-spectrum and orthogonal trans-

form, Jun. 2004.

[3] A. M. Aaron and B. Girod. Compression with side information using turbo

codes. In Proc. IEEE Data Compression Conf., Snowbird, Utah, 2002.

[4] A. M. Aaron and B. Girod. Wyner-Ziv video coding with low-encoder complex-

ity. In Proc. Picture Coding Symp., San Francisco, California, 2004.

[5] A. M. Aaron, S. Rane, and B. Girod. Wyner-Ziv video coding with hash-based

motion compensation at the receiver. In Proc. IEEE Internat. Conf. Image

Process., Singapore, 2004.

[6] A. M. Aaron, S. Rane, D. Rebollo-Monedero, and Bernd Girod. Systematic

lossy forward error protection for video waveforms. In Proc. IEEE Internat.

Conf. Image Process., Barcelona, Spain, 2003.

[7] A. M. Aaron, S. Rane, E. Setton, and B. Girod. Transform-domain Wyner-Ziv

codec for video. In SPIE Visual Communications and Image Process. Conf.,

San Jose, California, 2004.

[8] A. M. Aaron, S. Rane, R. Zhang, and B. Girod. Wyner-Ziv coding for video:

Applications to compression and error resilience. In Proc. IEEE Data Compres-

sion Conf., Snowbird, Utah, 2003.

102

BIBLIOGRAPHY 103

[9] A. M. Aaron, E. Setton, and Bernd Girod. Towards practical Wyner-Ziv coding

of video. In Proc. IEEE Internat. Conf. Image Process., Barcelona, Spain, 2003.

[10] A. M. Aaron, D. P. Varodayan, and B. Girod. Wyner-Ziv residual coding of

video. In Proc. Picture Coding Symp., Beijing, China, 2006.

[11] A. M. Aaron, R. Zhang, and B. Girod. Wyner-Ziv coding of motion video.

In Proc. Asilomar Conf. on Signals, Syst., Comput., Pacific Grove, California,

2002.

[12] R. Ahlswede and J. Korner. Source coding with side information and a converse

for degraded broadcast channels. IEEE Trans. Inform. Theory, 21(6):629–637,

November 1975.

[13] A. K. Al Jabri and S. Al-Issa. Zero-error codes for correlated information

sources. In Proc. IMA Internat. Conf. Cryptog. Coding, Cirencester, United

Kingdom, 1997.

[14] A. Amraoui. LTHC: LdpcOpt, visited on Nov. 15, 2005.

Available online at http://lthcwww.epfl.ch/research/ldpcopt.

[15] X. Artigas, E. Angeli, and L. Torres. Side information generation for multi-

view distributed video coding using a fusion approach. In Proc. Nordic Signal

Process. Symp., Reykjavik, Iceland, 2006.

[16] X. Artigas, J. Ascenso, M. Dalai, S. Klomp, D. Kubasov, and M. Ouaret. The

DISCOVER codec: Architecture, Techniques and Evaluation. In Proc. Picture

Coding Symp., Lisbon, Portugal, 2007.

[17] X. Artigas, S. Malinowski, C. Guillemot, and L. Torres. Overlapped quasi-

arithmetic codes for distributed video coding. In Proc. IEEE Internat. Conf.

Image Process., San Antonio, Texas, 2007.

[18] J. Ascenso, C. Brites, and F. Pereira. Improving frame interpolation with spatial

motion smoothing for pixel domain distributed video coding. In Proc. EURASIP

Conf. Speech Image Process., Multimedia Commun. Services, Smolenice, Slovak

Republic, 2005.

BIBLIOGRAPHY 104

[19] J. Ascenso, C. Brites, and F. Pereira. Content adaptive Wyner-Ziv video cod-

ing driven by motion activity. In Proc. IEEE Internat. Conf. Image Process.,

Altanta, Georgia, 2006.

[20] J. Ascenso, C. Brites, and F. Pereira. Design and performance of a novel low-

density parity-check code for distributed video coding. In Proc. IEEE Internat.

Conf. Image Process., San Diego, California, 2008.

[21] J. Ascenso and F. Pereira. Adaptive hash-based side information exploitation

for efficient Wyner-Ziv video coding. In Proc. IEEE Internat. Conf. Image

Process., San Antonio, Texas, 2007.

[22] J. Bajcsy and P. Mitran. Coding for the Slepian-Wolf problem with turbo codes.

In Proc. IEEE Global Commun. Conf., San Antonio, Texas, 2001.

[23] M. Barni, D. Papini, A. Abrardo, and E. Magli. Distributed source coding of

hyperspectral images. In Proc. Internat. Geoscience Remote Sensing Symp.,

Seoul, South Korea, 2005.

[24] R. J. Barron, B. Chen, and G. W. Wornell. The duality between information

embedding and source coding with side information and some applications.

IEEE Trans. Inform. Theory, 49(5):1159–1180, May 2003.

[25] L. E. Baum, T. Petrie, G. Soules, and N. Weiss. A maximization technique

occurring in the statistical analysis of probabilistic functions of Markov chains.

Ann. Math. Stat., 41(1):164–171, Oct. 1970.

[26] T. Berger. Multiterminal source coding. In Lecture Notes presented at CISM

Summer School, 1977.

[27] T. Berger and S. Tung. Encoding of correlated analog sources. In Proc. IEEE-

USSR Joint Workshop on Inform. Theory, Moscow, USSR, 1975.

[28] C. Berrou, A. Glavieux, and P. Thitimajshima. Near Shannon limit error-

correcting coding and decoding: Turbo-codes. In Proc. Internat. Conf. Com-

mun., Geneva, Switzerland, 1993.

BIBLIOGRAPHY 105

[29] R. B. Blizard. Convolutional coding for data compression. Martin Marietta

Corp., Denver Div., Rep. R-69-17, 1969.

[30] R. C. Bose and D. K. Ray-Chaudhuri. On a class of error correcting binary

group codes. Inf. Control, 3(1):68–79, Mar. 1960.

[31] C. Brites, J. Ascenso, and F. Pereira. Modeling correlation noise statistics at

decoder for pixel based Wyner-Ziv video coding. In Proc. Picture Coding Symp.,

Beijing, China, 2006.

[32] C. Brites, J. Ascenso, and F. Pereira. Studying temporal correlation noise

modeling for pixel based Wyner-Ziv video coding. In Proc. IEEE Internat.

Conf. Image Process., Altanta, Georgia, 2006.

[33] C. Brites and F. Pereira. Correlation noise modeling for efficient pixel and

transform domain WynerZiv video coding. IEEE Trans. Circuits Syst. Video

Technol., 18(9):1177–1190, Sept. 2008.

[34] J. Cardinal and G. Van Asche. Joint entropy-constrained multiterminal quanti-

zation. In Proc. Internat. Symp. Inform. Theory, Lausanne, Switzerland, 2002.

[35] D. M. Chen, D. P. Varodayan, M. Flierl, and B. Girod. Distributed stereo image

coding with improved disparity and noise estimation. In Proc. IEEE Internat.

Conf. Acoustics Speech Signal Process., Las Vegas, Nevada, 2008.

[36] D. M. Chen, D. P. Varodayan, M. Flierl, and B. Girod. Wyner-Ziv coding of

multiview images with unsupervised learning of disparity and Gray code. In

Proc. IEEE Internat. Conf. Image Process., San Diego, California, 2008.

[37] D. M. Chen, D. P. Varodayan, M. Flierl, and B. Girod. Wyner-Ziv coding of

multiview images with unsupervised learning of two disparities. In Proc. IEEE

Internat. Conf. Multimedia and Expo, Hannover, Germany, 2008.

[38] J. Chen, A. Khisti, D. M. Malioutov, and J. S. Yedidia. Distributed source

coding using serially-concatenated-accumulate codes. In IEEE Inform. Theory

Workshop, San Antonio, Texas, 2004.

BIBLIOGRAPHY 106

[39] Y. Chen, Y.-K. Wang, K. Ugur, M. M. Hannuksela, J. Lainema, and M. Gab-

bouj. The emerging MVC standard for 3D video services. EURASIP Adv.

Signal Process. J., 2009. doi:10.1155/2009/786015.

[40] N.-M. Cheung and A. Ortega. An efficient and highly parallel hyperspectral im-

agery compression scheme based on distributed source coding. In Proc. Asilomar

Conf. on Signals, Syst., Comput., Pacific Grove, California, 2006.

[41] N.-M. Cheung, C. Tang, A. Ortega, and C. S. Raghavendra. Efficient wavelet-

based predictive slepian-wolf coding for hyperspectral imagery. EURASIP Sig-

nal Process. J., 86(11):3180–3195, Nov. 2006.

[42] K. Chono, Y.-C. Lin, D. P. Varodayan, and B. Girod. Reduced-reference image

quality estimation using distributed source coding. In Proc. IEEE Internat.

Conf. Multimedia and Expo, Hannover, Germany, Jun. 2008.

[43] T. P. Coleman, A. H. Lee, M. Medard, and M. Effros. On some new approaches

to practical Slepian-Wolf compression inspired by channel coding. In Proc.

IEEE Data Compression Conf., Snowbird, Utah, 2004.

[44] M. Costa. Writing on dirty paper. IEEE Trans. Inform. Theory, 29(3):439–441,

May 1983.

[45] T. M. Cover. A proof of the data compression theorem of Slepian and Wolf for

ergodic sources. IEEE Trans. Inform. Theory, 21(2):226–228, Oct. 1975.

[46] T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley

& Sons, Inc., 1991.

[47] A. Dempster, N. Laird, and D. Rubin. Maximum likelihood from incomplete

data via the EM algorithm. J. Royal Stat. Soc., Series B, 39(1):1–38, 1977.

[48] C. Di, D. Proietti, I. E. Telatar, T. J. Richardson, and R. L. Urbanke. Finite-

length analysis of low-density parity-check codes on the binary erasure channel.

IEEE Trans. Inform. Theory, 48(6):1570–1579, Jun. 2002.

BIBLIOGRAPHY 107

[49] H. Dong, J. Lu, and Y. Sun. Distributed audio coding in wireless sensor net-

works. In Internat. Conf. Comput. Intelligence Security, Guangzhou, China,

2006.

[50] P. L. Dragotti and M. Gastpar. Distributed Source Coding. Elsevier, Inc., 2009.

[51] S. C. Draper. Successive structuring of source coding algorithms for data fusion,

buffering, and distribution in networks. Ph.D. dissertation, MIT, 2002.

[52] S. C. Draper, A. Khisti, E. Martinian, A. Vetro, and J. S. Yedidia. Secure

storage of fingerprint biometrics using Slepian-Wolf codes. In IEEE Inform.

Theory Applic. Workshop, San Diego, California, 2007.

[53] S. C. Draper, A. Khisti, E. Martinian, A. Vetro, and J. S. Yedidia. Using dis-

tributed source coding to secure fingerprint biometrics. In Proc. IEEE Internat.

Conf. Acoustics Speech Signal Process., Honolulu, Hawaii, 2007.

[54] S. C. Draper and G. W. Wornell. Side information aware coding strategies for

sensor networks. IEEE J. Select. Areas Commun., 22(6):966–976, Aug. 2004.

[55] F. Dufaux, M. Ouaret, and T. Ebrahimi. Recent advances in multiview dis-

tributed video coding. In Proc. Mobile Multimedia/Image Process. Military

Security Applic., Orlando, Florida, 2007.

[56] M. Effros and D. Muresan. Codecell contiguity in optimal fixed-rate and

entropy-constrained network scalar quantizers. In Proc. IEEE Data Compres-

sion Conf., Snowbird, Utah, 2002.

[57] M. Fleming and M. Effros. Network vector quantization. In Proc. IEEE Data

Compression Conf., Snowbird, Utah, 2001.

[58] M. Fleming, Q. Zhao, and M. Effros. Network vector quantization. IEEE Trans.

Inform. Theory, 50(8):1584–1604, Aug. 2004.

[59] M. Flierl and B. Girod. Coding of multi-view image sequences with video

sensors. In Proc. IEEE Internat. Conf. Image Process., Atlanta, Georgia, 2006.

BIBLIOGRAPHY 108

[60] M. Flierl and P. Vandergheynst. Video coding with motion-compensated tem-

poral transforms and side information. In Proc. IEEE Internat. Conf. Acoustics

Speech Signal Process., Philadelphia, Pennsylvania, 2005.

[61] T. Flynn and R. Gray. Encoding of correlated observations. IEEE Trans.

Inform. Theory, 33(6):773–787, Nov. 1987.

[62] T. Flynn and R. Gray. Correction to encoding of correlated observations. IEEE

Trans. Inform. Theory, 37(3):699, May 1991.

[63] T. Fujii and M. Tanimoto. Free viewpoint TV system based on ray-space repre-

sentation. In SPIE 3D TV Video Display Conf., Boston, Massachusetts, 2002.

[64] R. G. Gallager. Low-density parity-check codes. Ph.D. dissertation, MIT, 1963.

[65] J. Garcıa-Frıas. Compression of correlated binary sources using turbo codes.

IEEE Commun. Lett., 5(10):417–419, Oct. 2001.

[66] J. Garcıa-Frıas. Joint source-channel decoding of correlated sources over noisy

channels. In Proc. IEEE Data Compression Conf., Snowbird, Utah, 2001.

[67] J. Garcıa-Frıas. Decoding of low-density parity check codes over finite-state

binary Markov channels. IEEE Trans. Commun., 52(11):1840–1843, Nov. 2004.

[68] J. Garcıa-Frıas and Y. Zhao. Data compression of unknown single and correlated

binary sources using punctured turbo codes. In Proc. Allerton Conf. Commun.,

Contr. and Comput., Monticello, Illinois, 2001.

[69] J. Garcıa-Frıas and Y. Zhao. Compression of binary memoryless sources using

punctured turbo codes. IEEE Commun. Lett., 6(9):394–396, Sept. 2002.

[70] J. Garcıa-Frıas and W. Zhong. LDPC codes for asymmetric compression of

multi-terminal sources with hidden Markov correlation. In Proc. CTA Commun.

Networks Symp., College Park, Maryland, 2003.

[71] J. Garcıa-Frıas and W. Zhong. LDPC codes for compression of multi-terminal

sources with hidden Markov correlation. IEEE Commun. Lett., 7(3):115–117,

Mar. 2003.

BIBLIOGRAPHY 109

[72] M. Gastpar. On the Wyner-Ziv problem with two sources. In Proc. Internat.

Symp. Inform. Theory, Yokohama, Japan, 2003.

[73] M. Gastpar. On Wyner-Ziv networks. In Proc. Asilomar Conf. on Signals,

Syst., Comput., Pacific Grove, California, 2003.

[74] N. Gehrig and P. L. Dragotti. Symmetric and a-symmetric Slepian-Wolf codes

with systematic and nonsystematic linear codes. IEEE Commun. Lett., 9(1):61–

63, Jan. 2005.

[75] B. Girod, A. M. Aaron, S. Rane, and D. Rebollo-Monedero. Distributed video

coding. Proceedings IEEE, 93(1):71–83, Jan. 2005.

[76] S. W. Golomb. Shift Register Sequences. Holden-Day, 1967.

[77] M. Grangetto, E. Magli, and G. Olmo. Distributed arithmetic coding. IEEE

Commun. Lett., 11(11):883–885, Nov. 2007.

[78] C. Guillemot, F. Pereira, L. Torres, T. Ebrahimi, R. Leonardi, and J. Oster-

mann. Distributed monoview and multiview video coding. IEEE Signal Proc.

Magazine, 24(5):67–76, Sept. 2007.

[79] X. Guo, Y. Lu, F. Wu, W. Gao, and S. Li. Distributed multi-view video coding.

In SPIE Visual Communications and Image Process. Conf., San Jose, Califor-

nia, 2006.

[80] J. Ha and S. W. McLaughlin. Optimal puncturing of irregular low-density

parity-check codes. In Proc. IEEE Internat. Conf. on Commun., Anchorage,

Alaska, 2003.

[81] J. Hagenauer. Rate-compatible punctured convolutional codes (RCPC codes)

and their applications. IEEE Trans. Commun., 36(4):389–400, Apr. 1988.

[82] J. Hagenauer, J. Barros, and A. Schaeffer. Lossless turbo source coding with

decremental redundancy. In Proc. 5th ITG Conf. on Source and Channel Cod-

ing, Erlangen, Germany, 2004.

BIBLIOGRAPHY 110

[83] T. S. Han and K. Kobayashi. A unified achievable rate region for a general

class of multiterminal source coding systems. IEEE Trans. Inform. Theory,

26(3):277–288, May 1980.

[84] C. Heegard and T. Berger. Rate distortion when side information may be

absent. IEEE Trans. Inform. Theory, 31(6):727–734, Nov. 1985.

[85] M. E. Hellman. Convolutional source encoding. IEEE Trans. Inform. Theory,

21(6):651–656, Nov. 1975.

[86] A. Hocquenghem. Codes correcteurs d’erreurs. Chiffres (Paris), 2:147–156,

Sept. 1959.

[87] K. B. Housewright. Source coding studies for multiterminal systems. Ph.D.

dissertation, University of California, Los Angeles, 1979.

[88] X.-Y. Hu, E. Eleftheriou, and D. Arnold. Regular and irregular progressive

edge-growth Tanner graphs. IEEE Trans. Inform. Theory, 51(1):286–398, Jan.

2005.

[89] M. Isaka and M. P. C. Fossorier. High rate serially concatenated coding with

extended Hamming codes. IEEE Commun. Lett., 9(2):160–162, Feb. 2005.

[90] J. Jiang, D. He, and A. Jagmohan. Rateless Slepian-Wolf coding based on

rate adaptive low-density parity-check codes. In Proc. Internat. Symp. Inform.

Theory, Nice, France, 2007.

[91] B. Julesz. Binocular depth perception of computer generated patterns. Bell

Sys. Tech. J, 38:1001–1020, 1960.

[92] M. L. Kaiser. The STEREO mission: an overview. Advances Space Research,

36(8):1483–1488, Aug. 2005.

[93] R. Kawada, O. Sugimoto, A. Koike, M. Wada, and S. Matsumoto. Highly

precise estimation scheme for remote video PSNR using spread spectrum and

extraction of orthogonal transform coefficients. Electronics Commun. Japan

(Part I), 89(6):51–62, Jun. 2006.

BIBLIOGRAPHY 111

[94] J. Korner and K. Marton. How to encode the modulo-two sum of binary sources.

IEEE Trans. Inform. Theory, 25(2):219–221, March 1979.

[95] P. Koulgi, E. Tuncel, E. Rengunathan, and K. Rose. On zero-error source

coding of correlated sources. IEEE Trans. Inform. Theory, 49(11):2856–2873,

Nov. 2003.

[96] P. Koulgi, E. Tuncel, E. Rengunathan, and K. Rose. On zero-error source coding

with decoder side information. IEEE Trans. Inform. Theory, 49(1):99–111, Jan.

2003.

[97] F. R. Kschischang, B. J. Frey, and H.-A. Loeliger. Factor graphs and the sum-

product algorithm. IEEE Trans. Inform. Theory, 47(2):498–519, Feb. 2001.

[98] D. Kubasov, J. Nayak, and C. Guillemot. Optimal reconstruction in Wyner-Ziv

video coding with multiple side information. In Proc. IEEE Internat. Workshop

Multimedia Signal Process., Chania, Greece, 2007.

[99] J. Kusuma, L. Doherty, and K. Ramchandran. Distributed compression for

sensor networks. In Proc. IEEE Internat. Conf. Image Process., Thessaloniki,

Greece, 2001.

[100] C. F. Lan, A. Liveris, K. Narayanan, Z. Xiong, and C. Georghiades. Slepian-

Wolf coding of multiple M-ary sources using LDPC codes. In Proc. IEEE Data


[101] J. Li and H. Alqamzi. An optimal distributed and adaptive source coding

strategy using rate-compatible punctured convolutional codes. In Proc. IEEE

Internat. Conf. Acoustics Speech Signal Process., Philadelphia, Pennsylvania,

2005.

[102] J. Li, K. R. Narayanan, and C. N. Georghiades. Product accumulate codes:

a class of codes with near-capacity performance and low decoding complexity.

IEEE Trans. Inform. Theory, 50(1):31–46, Jan. 2004.

[103] X. Li. Distributed coding of multispectral images: a set theoretic approach. In

Proc. IEEE Internat. Conf. Image Process., Singapore, 2004.

BIBLIOGRAPHY 112

[104] Z. Li, Y.-C. Lin, D. P. Varodayan, P. Baccichet, and B. Girod. Distortion-

aware retransmission and concealment of video packets using a Wyner-Ziv-

coded thumbnail. In Proc. IEEE Internat. Workshop Multimedia Signal Pro-

cess., Cairns, Australia, Oct. 2008.

[105] Y.-C. Lin, D. P. Varodayan, T. Fink, E. Bellers., and B. Girod. Authenticating

contrast and brightness adjusted images using distributed source coding and

expectation maximization. In Proc. IEEE Internat. Conf. Multimedia and Expo,

Hannover, Germany, Jun. 2008.

[106] Y.-C. Lin, D. P. Varodayan, T. Fink, E. Bellers, and B. Girod. Localization of

tampering in contrast and brightness adjusted images using distributed source

coding and expectation maximization. In Proc. IEEE Internat. Conf. Image

Process., San Diego, California, Oct. 2008.

[107] Y.-C. Lin, D. P. Varodayan, and B. Girod. Image authentication and tampering

localization using distributed source coding. In Proc. IEEE Internat. Workshop

Multimedia Signal Process., Chania, Greece, Oct. 2007.

[108] Y.-C. Lin, D. P. Varodayan, and B. Girod. Image authentication based on

distributed source coding. In Proc. IEEE Internat. Conf. Image Process., San

Antonio, Texas, Sept. 2007.

[109] Y.-C. Lin, D. P. Varodayan, and B. Girod. Spatial models for localization of

image tampering using distributed source codes. In Proc. Picture Coding Symp.,

Lisbon, Portugal, Nov. 2007.

[110] Y.-C. Lin, D. P. Varodayan, and B. Girod. Authenticating cropped and resized

images using distributed source coding and expectation maximization. In SPIE

Electronic Imaging Media Forensics Security XI, San Jose, California, Jan. 2009.

[111] Y.-C. Lin, D. P. Varodayan, and B. Girod. Distributed source coding authenti-

cation of images with affine warping. In Proc. IEEE Internat. Conf. Acoustics

Speech Signal Process., Taipei, Taiwan, Apr. 2009.

BIBLIOGRAPHY 113

[112] Y.-C. Lin, D. P. Varodayan, and B. Girod. Distributed source coding authen-

tication of images with contrast and brightness adjustment and affine warping.

In Proc. Picture Coding Symp., Chicago, Illinois, May 2009.

[113] Y.-C. Lin, D. P. Varodayan, and B. Girod. Video quality monitoring for mo-

bile multicast peers using distributed source coding. In Proc. Internat. Mobile

Multimedia Commun. Conf., London, United Kingdom, Sept. 2009.

[114] T. Linder, R. Zamir, and K. Zeger. On source coding with side information for

general distortion measures. In Proc. Internat. Symp. Inform. Theory, Cam-

bridge, Massachusetts, 1998.

[115] T. Linder, R. Zamir, and K. Zeger. On source coding with side-information-

dependent distortion measures. IEEE Trans. Inform. Theory, 46(7):2697–2704,

Nov. 2000.

[116] Z. Liu, S. Cheng, A. Liveris, and Z. Xiong. Slepian-Wolf coded nested quanti-

zation SWC-NQ for Wyner-Ziv coding: performance analysis and code design.

In Proc. IEEE Data Compression Conf., Snowbird, Utah, 2004.

[117] A. Liveris, Z. Xiong, and C. Georghiades. Compression of binary sources with

side information at the decoder using LDPC codes. IEEE Commun. Lett.,

6(10):440–442, Oct. 2002.

[118] A. Liveris, Z. Xiong, and C. Georghiades. Compression of binary sources with

side information at the decoder using LDPC codes. In Proc. IEEE Global

Commun. Conf., Taipei, Taiwan, 2002.

[119] A. Liveris, Z. Xiong, and C. Georghiades. Joint source-channel coding of bi-

nary sources with side information at the decoder using IRA codes. In Proc.

IEEE Internat. Workshop Multimedia Signal Process., St. Thomas, U.S. Virgin

Islands, 2002.

[120] S. P. Lloyd. Least squares quantization in PCM. IEEE Trans. Inform. Theory,

28(2):129–137, Mar. 1982.

[121] D. J. C. MacKay. Good error-correcting codes based on very sparse matrices.

IEEE Trans. Inform. Theory, 45(2):399–431, Mar. 1999.

BIBLIOGRAPHY 114

[122] D. J. C. MacKay. Source code for progressive edge growth parity-check matrix

construction, 2004.

[123] D. J. C. MacKay, S. T. Wilson, and M. C. Davey. Comparison of constructions

of irregular Gallager codes. IEEE Trans. Commun., 47(10):1449–1454, Oct.

1999.

[124] A. Majumdar, K. Ramchandran, and I. Kozintsev. Distributed coding for wire-

less audio sensors. In Proc. IEEE Applic. Signal Process. Audio Acoustics Work-

shop, New Paltz, New York, 2003.

[125] S. Malinowski, X. Artigas, C. Guillemot, and L. Torres. Distributed coding

using punctured quasi-arithmetic codes for memory and memoryless sources.

IEEE Trans. Signal Process. Accepted.

[126] E. Martinian, S. Yekhanin, and J. S. Yedidia. Secure biometrics via syndromes.

In Proc. Allerton Conf. Commun., Contr. and Comput., Monticello, Illinois,

2005.

[127] W. Matusik and H. Pfister. 3D TV: a scalable system for real-time acquisition,

transmission, and autostereoscopic display of dynamic scenes. ACM Trans.

Graphics, 23(3):814–824, Aug. 2004.

[128] P. Mitran and J. Bajcsy. Near Shannon-limit coding for the Slepian-Wolf prob-

lem. In Proc. Biennial Symp. Commun., Kingston, Ontario, Canada, 2002.

[129] P. Mitran and J. Bajcsy. Turbo source coding: a noise-robust approach to data

compression. In Proc. IEEE Data Compression Conf., Snowbird, Utah, 2002.

[130] D. Muresan and M. Effros. Quantization as histogram segmentation: glob-

ally optimal scalar quantizer design in network systems. In Proc. IEEE Data


[131] NASA. NASA - STEREO Mission, visited on Mar. 9, 2010.

Available online at http://www.nasa.gov/stereo.

BIBLIOGRAPHY 115

[132] L. Natario, C. Brites, J. Ascenso, and F. Pereira. Extrapolating side infor-

mation for low-delay pixel-domain distributed video coding. In Proc. Internat.

Workshop Very Low Bitrate Video Coding, Sardinia, Italy, 2005.

[133] M. Ouaret, F. Dufaux, and T. Ebrahimi. Fusion-based multiview distributed

video coding. In Proc. ACM Internat. Workshop Video Surveillance Sensor

Networks, Santa Barbara, California, 2006.

[134] J. Pearl. Reverend Bayes on inference engines: A distributed hierarchical ap-

proach. In Proc. AAAI Conf. Artificial Intelligence, Pittsburgh, Pennsylvania,

1982.

[135] F. Pereira, L. Torres, C. Guillemot, T. Ebrahimi, R. Leonardi, and S. Klomp.

Distributed video coding: selecting the most promising application scenarios.

EURASIP Signal Process.: Image Commun. J, 23(5):339–352, Jun. 2008.

[136] H. Pishro-Nik and F. Fekri. Results on punctured LDPC codes. In IEEE

Inform. Theory Workshop, San Antonio, Texas, 2004.

[137] S. S. Pradhan, J. Chou, and K. Ramchandran. Duality between source coding

and channel coding and its extension to the side information case. IEEE Trans.

Inform. Theory, 49(5):1181–1203, May 2003.

[138] S. S. Pradhan, J. Kusuma, and K. Ramchandran. Distributed compression in

a dense microsensor network. IEEE Signal Process. Mag., 19(2):51–60, Mar.

2002.

[139] S. S. Pradhan and K. Ramchandran. Distributed source coding using syndromes

(DISCUS): design and construction. In Proc. IEEE Data Compression Conf.,

Snowbird, Utah, 1999.

[140] S. S. Pradhan and K. Ramchandran. Distributed source coding: symmetric

rates and applications to sensor networks. In Proc. IEEE Data Compression

Conf., Snowbird, Utah, 2000.

[141] S. S. Pradhan and K. Ramchandran. Geometric proof of rate-distortion function

of Gaussian sources with side information at the decoder. In Proc. Internat.

Symp. Inform. Theory, Sorrento, Italy, 2000.

BIBLIOGRAPHY 116

[142] S. S. Pradhan and K. Ramchandran. Group-theoretic construction and analysis

of generalized coset codes for symmetric/asymmetric distributed source coding.

In Proc. Conf. Inform. Sciences Syst., Princeton, New Jersey, 2000.

[143] S. S. Pradhan and K. Ramchandran. Enhancing analog image transmission

systems using digital side information: a new wavelet-based image coding

paradigm. In Proc. IEEE Data Compression Conf., Snowbird, Utah, 2001.

[144] S. S. Pradhan and K. Ramchandran. Distributed source coding using syndromes

(DISCUS): design and construction. IEEE Trans. Inform. Theory, 49(3):626–

643, Mar. 2003.

[145] G. Prandi, G. Valenzise, M. Tagliasacchi, and A. Sarti. Detection and identifica-

tion of sparse audio tampering using distributed source coding and compressive

sensing techniques. In Proc. Internat. Conf. Digital Audio Effects, Espoo, Fin-

land, Sept. 2008.

[146] R. Puri, A. Majumdar, and K. Ramchandran. PRISM: A video coding

paradigm with motion estimation at the decoder. IEEE Trans. Image Process.,

16(10):24362448, Oct. 2007.

[147] R. Puri and K. Ramchandran. PRISM: a new robust video coding architecture

based on distributed compression principles. In Proc. Allerton Conf. Commun.,

Contr. and Comput., Monticello, Illinois, 2002.

[148] R. Puri and K. Ramchandran. PRISM: a ‘reversed’ multimedia coding

paradigm. In Proc. IEEE Internat. Conf. Image Process., Barcelona, Spain,

2003.

[149] R. Puri and K. Ramchandran. PRISM: an uplink-friendly multimedia coding

paradigm. In Proc. IEEE Internat. Conf. Acoustics Speech Signal Process.,

Hong Kong, China, 2003.

[150] S. Rane. Systematic lossy error protection of video signals. Ph.D. dissertation,

Stanford University, 2007.

BIBLIOGRAPHY 117

[151] S. Rane, A. M. Aaron, and B. Girod. Systematic lossy forward error protection

for error resilient digital video broadcasting. In SPIE Visual Communications

and Image Process. Conf., San Jose, California, 2004.

[152] S. Rane, P. Baccichet, and B. Girod. Systematic lossy error protection of video

signals. IEEE Trans. Circuits Syst. Video Technol., 18(10):1347–1360, Nov.

2008.

[153] D. Rebollo-Monedero. Quantization and transforms for distributed source cod-

ing. Ph.D. dissertation, Stanford University, 2007.

[154] D. Rebollo-Monedero, A. M. Aaron, and B. Girod. Transforms for high-rate

distributed source coding. In Proc. Asilomar Conf. on Signals, Syst., Comput.,

Pacific Grove, California, 2003.

[155] D. Rebollo-Monedero and B. Girod. Design of optimal quantizers for distributed

coding of noisy sources. In Proc. IEEE Internat. Conf. Acoustics Speech Signal

Process., Philadelphia, Pennsylvania, 2005.

[156] D. Rebollo-Monedero and B. Girod. Network distributed quantization. In Proc.

IEEE Inform. Theory Workshop, Tahoe City, California, 2007.

[157] D. Rebollo-Monedero, S. Rane, A. M. Aaron, and B. Girod. High-rate quanti-

zation and transform coding with side information at the decoder. EURASIP

Signal Process. J., 86(11):3160–3179, Nov. 2006.

[158] D. Rebollo-Monedero, S. Rane, and B. Girod. Wyner-Ziv quantization and

transform coding of noisy sources at high rates. In Proc. Asilomar Conf. on

Signals, Syst., Comput., Pacific Grove, California, 2004.

[159] D. Rebollo-Monedero, R. Zhang, and B. Girod. Design of optimal quantizers for

distributed source coding. In Proc. IEEE Data Compression Conf., Snowbird,

Utah, 2003.

[160] T. J. Richardson, A. Shokrollahi, and R. L. Urbanke. Design of capacity-

approaching irregular low-density parity-check codes. IEEE Trans. Inform.

Theory, 47(2):619–637, Feb. 2001.

BIBLIOGRAPHY 118

[161] T. J. Richardson and R. L. Urbanke. The capacity of low-density parity check

codes under message-passing decoding. IEEE Trans. Inform. Theory, 47(2):599–

618, Feb. 2001.

[162] T. J. Richardson and R. L. Urbanke. Modern Coding Theory. Cambridge

University Press, 2008.

[163] A. Roumy, K. Lajnef, and C. Guillemot. Rate-adaptive turbo syndrome scheme

for Slepian-Wolf coding. In Proc. Asilomar Conf. on Signals, Syst., Comput.,

Pacific Grove, California, 2007.

[164] D. N. Rowitch and L. B. Milstein. On the performance of hybrid FEC/ARQ

systems using rate compatible punctured turbo (RCPT) codes. IEEE Trans.

Commun., 48(6):948–959, Jun. 2000.

[165] O. Roy and M. Vetterli. Distributed spatial audio coding in wireless hearing

aids. In Proc. IEEE Applic. Signal Process. Audio Acoustics Workshop, New

Paltz, New York, 2007.

[166] M. Sartipi and F. Fekri. Distributed source coding in wireless sensor networks

using LDPC coding: the entire Slepian-Wolf rate region. In Proc. IEEE Wireless

Commun. Networking Conf., New Orleans, Louisiana, 2005.

[167] M. Sartipi and F. Fekri. Distributed source coding using short to moderate

length rate-compatible LDPC codes: the entire Slepian-Wolf rate region. IEEE

Trans. Commun., 56(3):400–411, Mar. 2008.

[168] D. Schonberg, S. S. Pradhan, and K. Ramchandran. LDPC codes can achieve

the Slepian-Wolf bound for general binary sources. In Proc. Allerton Conf.

Commun., Contr. and Comput., Monticello, Illinois, 2002.

[169] D. Schonberg, S. S. Pradhan, and K. Ramchandran. Distributed code construc-

tion for the entire Slepian-Wolf rate region for arbitralily correlated sources.

In Proc. Asilomar Conf. on Signals, Syst., Comput., Pacific Grove, California,

2003.

BIBLIOGRAPHY 119

[170] D. Schonberg, S. S. Pradhan, and K. Ramchandran. Distributed code construc-

tion for the entire Slepian-Wolf rate region for arbitralily correlated sources. In

Proc. IEEE Data Compression Conf., Snowbird, Utah, 2004.

[171] A. Sehgal and N. Ahuja. Robust predictive coding and the Wyner-Ziv problem.

In Proc. IEEE Data Compression Conf., Snowbird, Utah, 2003.

[172] A. Sehgal, A. Jagmohan, and N. Ahuja. A causal state-free video encoding

paradigm. In Proc. IEEE Internat. Conf. Image Process., Barcelona, Spain,

2003.

[173] S. D. Servetto. Lattice quantization with side information. In Proc. IEEE Data


[174] S. Sesia, G. Caire, and G. Vivier. Incremental redundancy hybrid ARQ schemes

based on low-density parity-check codes. IEEE Trans. Commun., 52(8):1311–

1321, Aug. 2004.

[175] C. E. Shannon. A mathematical theory of communication. Bell Sys. Tech. J,

27:379–423, 623–656, 1948.

[176] G. Simonyi. On Witsenhausen’s zero-error rate for multiple sources. IEEE

Trans. Inform. Theory, 49(12):3258–3261, Dec. 2003.

[177] D. Slepian and J. K. Wolf. Noiseless coding of correlated information sources.

IEEE Trans. Inform. Theory, 19(4):471–480, Jul. 1973.

[178] B. Song, O. Bursalioglu, A. Roy-Chowdhury, and E. Tuncel. Towards a multi-

terminal video compression algorithm using epipolar geometry. In Proc. IEEE

Internat. Conf. Acoustics Speech Signal Process., Toulouse, France, 2006.

[179] B. Song, E. Tuncel, and A. Roy-Chowdhury. Towards a multi-terminal video

compression algorithm by integrating distributed source coding with geometri-

cal constraints. J. Multimedia, 2(3):9–16, Jun. 2007.

[180] V. Stankovic, A. Liveris, Z. Xiong, and C. Georghiades. Design of Slepian-Wolf

codes by channel code partitioning. In Proc. IEEE Data Compression Conf.,


BIBLIOGRAPHY 120

[181] V. Stankovic, A. Liveris, Z. Xiong, and C. Georghiades. On code design for

the Slepian-Wolf problem and lossless multiterminal networks. IEEE Trans.

Inform. Theory, 52(4):1495–1507, Apr. 2006.

[182] Y. Steinberg and N. Merhav. On successive refinement for the Wyner-Ziv prob-

lem. IEEE Trans. Inform. Theory, 50(8):1636–1654, Aug. 2004.

[183] Y. Steinberg and N. Merhav. On hierarchical joint source-channel coding with

degraded side information. IEEE Trans. Inform. Theory, 52(3):886–903, Mar

2006.

[184] J. K. Su, J. J. Eggers, and B. Girod. Illustration of the duality between channel

coding and rate distortion with side information. In Proc. Asilomar Conf. on

Signals, Syst., Comput., Pacific Grove, California, 2000.

[185] Y. Sutcu, S. Rane, J. S. Yedidia, S. C. Draper, and A. Vetro. Feature extraction

for a Slepian-Wolf biometric system using LDPC codes. In Proc. Internat. Symp.

Inform. Theory, Toronto, Ontario, Canada, 2008.

[186] Y. Sutcu, S. Rane, J. S. Yedidia, S. C. Draper, and A. Vetro. Feature transfor-

mation for a Slepian-Wolf biometric system based on error correcting codes. In

Proc. Comput. Vision Pattern Recog. Biometrics Workshop, Anchorage, Alaska,

2008.

[187] M. Tagliasacchi, G. Prandi, and S. Tubaro. Symmetric distributed coding of

stereo video sequences. In Proc. IEEE Internat. Conf. Image Process., San

Antonio, Texas, 2007.

[188] M. Tagliasacchi, G. Valenzise, M. Naccari, and S. Tubaro. Reduced-reference

video quality assessment using distributed source coding. Springer Multimedia

Tools Applic. Accepted.

[189] M. Tagliasacchi, G. Valenzise, and S. Tubaro. Localization of sparse image tam-

pering via random projections. In Proc. IEEE Internat. Conf. Image Process.,

San Diego, California, Oct. 2008.

BIBLIOGRAPHY 121

[190] P. Tan and J. Li. Enhancing the robustness of distributed compression using

ideas from channel coding. In Proc. IEEE Global Commun. Conf., St. Louis,

Missouri, 2005.

[191] P. Tan and J. Li. A practical and optimal symmetric Slepian-Wolf compres-

sion strategy using syndrome formers and inverse syndrome formers. In Proc.

Allerton Conf. Commun., Contr. and Comput., Monticello, Illinois, 2005.

[192] C. Tang, N.-M. Cheung, A. Ortega, and C. S. Raghavendra. Efficient inter-

band prediction and wavelet based compression for hyperspectral imagery: A

distributed source coding approach. In Proc. IEEE Data Compression Conf.,


[193] M. Tanimoto. Free viewpoint television – FTV. In Proc. Picture Coding Symp.,

San Francisco, California, 2004.

[194] M. Tanimoto. FTV (free viewpoint television) creating ray-based image engi-

neering. In Proc. IEEE Internat. Conf. Image Process., Genoa, Italy, 2005.

[195] M. Tanimoto. Tanimoto Lab. Dept. of Info. Elec. Nagoya University, visited on

Mar. 9, 2010.

Available online at http://www.tanimoto.nuee.nagoya-u.ac.jp/english.

[196] M. Tanimoto, T. Fujii, and N. Fukushima. 1D parallel test sequences for MPEG-

FTV (M15378). In ISO/IEC JTC1/SC29/WG11, Archamps, France, 2008.

[197] S. ten Brink. Designing iterative decoding schemes with the extrinsic infor-

mation transfer chart. AEU Int. J. Electron. Commun., 54(6):389–398, Dec.

2000.

[198] G. Toffetti, M. Tagliasacchi, M. Marcon, A. Sarti, S. Tubaro, and K. Ram-

chandran. Image compression in a multi-camera system based on a distributed

source coding approach. In Proc. Euro. Signal Process. Conf., Antalya, Turkey,

2005.

[199] V. Toto-Zarasoa, A. Roumy, and C. Guillemot. Rate-adaptive codes for the

entire Slepian-Wolf region and arbitrarily correlated sources. In Proc. IEEE

Internat. Conf. Acoustics Speech Signal Process., Las Vegas, Nevada, 2008.

BIBLIOGRAPHY 122

[200] E. Tuncel. Kraft inequality and zero-error source coding with decoder side

information. IEEE Trans. Inform. Theory, 53(12):4810–4816, Dec. 2007.

[201] S. Tung. Multiterminal source coding. Ph.D. dissertation, Cornell University,

1977.

[202] G. Valenzise, M. Naccari, M. Tagliasacchi, and S. Tubaro. Reduced-reference

estimation of channel-induced video distortion using distributed source coding.

In Proc. ACM Multimedia, Vancouver, British Columbia, Canada, Oct. 2008.

[203] G. Valenzise, G. Prandi, and M. Tagliasacchi. Identification of sparse audio

tampering using distributed source coding and compressive sensing techniques.

EURASIP Image Video Process. J., 2009. doi: 10.1155/2009/158982.

[204] D. P. Varodayan, A. M. Aaron, and B. Girod. Rate-adaptive distributed source

coding using low-density parity-check codes. In Proc. Asilomar Conf. on Sig-

nals, Syst., Comput., Pacific Grove, California, 2005.

[205] D. P. Varodayan, A. M. Aaron, and B. Girod. Rate-adaptive codes for dis-

tributed source coding. EURASIP Signal Process. J., 86(11):3123–3130, Nov.

2006.

[206] D. P. Varodayan, D. M. Chen, M. Flierl, and B. Girod. Wyner-Ziv coding of

video with unsupervised motion vector learning. EURASIP Signal Process.:

Image Commun. J, 23(5):369–378, Jun. 2008.

[207] D. P. Varodayan, Y.-C. Lin, and B. Girod. Audio authentication based on

distributed source coding. In Proc. IEEE Internat. Conf. Acoustics Speech

Signal Process., Las Vegas, Nevada, Apr. 2008.

[208] D. P. Varodayan, Y.-C. Lin, A. Mavlankar, M. Flierl, and B. Girod. Wyner-Ziv

coding of stereo images with unsupervised learning of disparity. In Proc. Picture

Coding Symp., Lisbon, Portugal, 2007.

[209] D. P. Varodayan, A. Mavlankar, M. Flierl, and B. Girod. Distributed coding of

random dot stereograms with unsupervised learning of disparity. In Proc. IEEE

BIBLIOGRAPHY 123

Internat. Workshop Multimedia Signal Process., Victoria, British Columbia,

Canada, 2006.

[210] D. P. Varodayan, A. Mavlankar, M. Flierl, and B. Girod. Distributed grayscale

stereo image coding with unsupervised learning of disparity. In Proc. IEEE

Data Compression Conf., Snowbird, Utah, 2007.

[211] A. Wagner and V. Anantharam. An infeasibility result for the multiterminal

source-coding problem. IEEE Trans. Inform. Theory. Submitted.

[212] A. Wagner and V. Anantharam. An improved outer bound for the multiterminal

source coding problem. In Proc. Internat. Symp. Inform. Theory, Adelaide,

Australia, 2005.

[213] A. Wagner, S. Tavildar, and P. Viswanath. Rate-region of the quadratic gaussian

two-encoder source-coding problem. IEEE Trans. Inform. Theory, 54(5):1938–

1961, May 2008.

[214] J. Wang, A. Majumdar, and K. Ramchandran. Robust transmission over a lossy

network using a distributed source coded auxiliary channel. In Proc. Picture

Coding Symp., San Francisco, California, 2004.

[215] X. Wang and M. Orchard. Design of trellis codes for source coding with side

information at the decoder. In Proc. IEEE Data Compression Conf., Snowbird,

Utah, 2001.

[216] S. B. Wicker. Error Control Systems for Digital Communication and Storage.

Prentice Hall, 1995.

[217] T. Wiegand, G. Sullivan, G. Bjøntegaard, and A. Luthra. Overview of the

H.264/AVC video coding standard. IEEE Trans. Circuits Syst. Video Technol.,

13(7):560–576, Jul. 2003.

[218] B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams,

M. Levoy, and M. Horowitz. High performance imaging using large camera

arrays. ACM Trans. Graph., 24(3):765–776, Jul. 2005.

BIBLIOGRAPHY 124

[219] B. Wilburn, M. Smulski, H.-H. Lee, and M. Horowitz. The light field video

camera. In SPIE Media Processors Conf., San Jose, California, 2002.

[220] H. S. Witsenhausen. The zero-error side information problem and chromatic

numbers. IEEE Trans. Inform. Theory, 22(5):592–593, Sept. 1976.

[221] H. S. Witsenhausen. Indirect rate-distortion problems. IEEE Trans. Inform.

Theory, 26(5):518–521, Sept. 1980.

[222] H. S. Witsenhausen and A. D. Wyner. Interframe coder for video signals. US

Patent 4191970, Tech. Rep., Nov. 1980.

[223] A. D. Wyner. Recent results in the Shannon theory. IEEE Trans. Inform.

Theory, 20(1):2–10, Jan. 1974.

[224] A. D. Wyner. On source coding with side information at the decoder. IEEE

Trans. Inform. Theory, 21(5):294–300, May 1975.

[225] A. D. Wyner. The rate-distortion function for source coding with side informa-

tion at the decoder-II: general sources. Inf. Control, 38(1):60–80, Jul. 1978.

[226] A. D. Wyner and J. Ziv. The rate-distortion function for source coding with

side information at the decoder. IEEE Trans. Inform. Theory, 22(1):1–10, Jan.

1976.

[227] Z. Xiong, A. Liveris, and S. Cheng. Distributed source coding for sensor net-

works. IEEE Signal Process. Mag., 21(5):80–94, Mar. 2004.

[228] Z. Xiong, A. Liveris, S. Cheng, and Z. Liu. Nested quantization and Slepian-

Wolf coding: a Wyner-Ziv coding paradigm for i.i.d. sources. In Proc. IEEE

Workshop on Statistical Signal Process., St. Louis, MO, 2003.

[229] Q. Xu and Z. Xiong. Layered Wyner-Ziv video coding. IEEE Trans. Image

Process., 15(12):3791–3803, Dec. 2006.

[230] T. Yamada, Y. Miyamoto, and M. Serizawa. No-reference video quality esti-

mation based on error-concealment effectiveness. In Proc. IEEE Packet Video

Conf., Lausanne, Switzerland, Nov. 2007.

BIBLIOGRAPHY 125

[231] T. Yamada, Y. Miyamoto, M. Serizawa, and H. Harasaki. Reduced-reference

based video quality metrics using representative luminance values. In Proc. In-

ternat. Video Process. Quality Metrics Consumer Electronics Workshop, Scotts-

dale, Arizona, Jan. 2007.

[232] H. Yamamoto and K. Itoh. Source coding theory for multiterminal communi-

cation systems with a remote source. Trans. IECE Japan, E63:700–706, Oct.

1980.

[233] Y. Yan and T. Berger. Zero-error instantaneous coding of correlated sources

with length constraints is NP-complete. IEEE Trans. Inform. Theory,

52(4):1705–1708, Apr. 2006.

[234] M. Yang, W. E. Ryan, and Y. Li. Design of efficiently encodable moderate-

length high-rate irregular LDPC codes. IEEE Trans. Commun., 52(4):564–571,

Apr. 2004.

[235] Y. Yang, S. Cheng, Z. Xiong, and W. Zhao. Wyner-Ziv coding based on TCQ

and LDPC codes. In Proc. Asilomar Conf. on Signals, Syst., Comput., Pacific

Grove, California, 2003.

[236] Y. Yang, V. Stankovic, Z. Xiong, and W. Zhao. Two-terminal video coding.

IEEE Trans. Image Process., 18(3):534551, Mar. 2009.

[237] Y. Yang, V. Stankovic, W. Zhao, and Z. Xiong. Multiterminal video coding. In

Proc. IEEE Internat. Conf. Image Process., San Antonio, Texas, 2007.

[238] C. Yeo and K. Ramchandran. Robust distributed multiview video compres-

sion for wireless camera networks. In SPIE Visual Communications and Image

Process. Conf., San Jose, California, 2007.

[239] R. Zamir. The rate loss in the Wyner-Ziv problem. IEEE Trans. Inform.

Theory, 42(6):2073–2084, Nov. 1996.

[240] R. Zamir and S. Shamai. Nested linear/lattice codes for Wyner-Ziv encoding.

In IEEE Inform. Theory Workshop, Killarney, Ireland, 1998.

BIBLIOGRAPHY 126

[241] R. Zamir, S. Shamai, and U. Erez. Nested linear/lattice codes for structured

multiterminal binning. IEEE Trans. Inform. Theory, 48(6):1250–1276, Jun.

2002.

[242] Q. Zhao and M. Effros. Optimal code design for lossless and near lossless source

coding in multiple access networks. In Proc. IEEE Data Compression Conf.,


[243] Y. Zhao and J. Garcıa-Frıas. Data compression of correlated non-binary sources

using punctured turbo codes. In Proc. IEEE Data Compression Conf., Snow-

bird, Utah, 2002.

[244] Y. Zhao and J. Garcıa-Frıas. Joint estimation and data compression of corre-

lated non-binary sources using punctured turbo codes. In Proc. Conf. Inform.

Sciences Syst., Princeton, New Jersey, 2002.

[245] Y. Zhao and J. Garcıa-Frıas. Turbo codes for symmetric compression of corre-

lated binary sources with hidden Markov correlation. In Proc. CTA Commun.

Networks Symp., College Park, Maryland, 2003.

[246] Y. Zhao and J. Garcıa-Frıas. Joint estimation and compression of correlated

non-binary sources using punctured turbo codes. IEEE Trans. Commun.,

53(3):385–390, Mar. 2005.

[247] Y. Zhao and J. Garcıa-Frıas. Turbo compression/joint source channel coding

of correlated binary sources with hidden Markov correlation. EURASIP Signal

Process. J., 86(11):3115–3122, Nov. 2006.

[248] W. Zhong and J. Garcıa-Frıas. Compression of non-binary sources using LDPC

codes. In Proc. Conf. Inform. Sciences Syst., Baltimore, Maryland, 2005.

[249] G. Zhu and F. Alajaji. Turbo codes for nonuniform memoryless sources over

noisy channels. IEEE Commun. Lett., 6(2):64–66, Feb. 2002.

[250] X. Zhu, A. M. Aaron, and B. Girod. Distributed compression for large cam-

era arrays. In Proc. IEEE Workshop on Statistical Signal Process., St. Louis,

Missouri, 2003.

adaptive distributed source coding a...

Documents