communicating correlated sources over user...

Communicating Correlated Sources over 2−user MAC

Arun Padakandla

Center for Science of Information, Purdue University

June 28, 2017

Communicating distributed correlated sources over channelsPart 2 : Joint source-channel coding technique

Arun Padakandla

Center for Science of Information, Purdue University

June 28, 2017

Problem Setup : Transmitting Correlated sources over 2−user MAC

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Shannon-theoretic study

?? Optimal coding tech. ??

?? Necessary and sufficient conditions in terms of WS1S2 , WY |X1X2. ??

Plain Vanilla Lossless source-channel coding over 2-user MAC.

1. A new coding tech.

2. Characterize performance, derive new sufficient conditions.

Problem Setup : Transmitting Correlated sources over 2−user MAC

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Shannon-theoretic study

?? Optimal coding tech. ??

?? Necessary and sufficient conditions in terms of WS1S2 , WY |X1X2. ??

Plain Vanilla Lossless source-channel coding over 2-user MAC.

1. A new coding tech.

2. Characterize performance, derive new sufficient conditions.

Problem Setup : Transmitting Correlated sources over 2−user MAC

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Shannon-theoretic study

?? Optimal coding tech. ??

?? Necessary and sufficient conditions in terms of WS1S2 , WY |X1X2. ??

Plain Vanilla Lossless source-channel coding over 2-user MAC.

1. A new coding tech.

2. Characterize performance, derive new sufficient conditions.

Big picture : Where is this going?

General multi-terminal problems : Seems NO single-letter (S-L) expr. forcapacity.

Example : Our fauvorite point-to-point chnl (PTP).

WY|X

X Y

Capacity [Shannon ’48] = suppX

I(X;Y ), and NOT (1)

Capacity [Shannon ’48] = supn≥1

suppnX

1

nI(Xn;Y n)

suppXand supn≥1 suppn

X: world of difference .

Associated with (1) is a single-letter (S-L) coding tech.

Big picture : Where is this going?

General multi-terminal problems : Seems NO single-letter (S-L) expr. forcapacity.

Example : Our fauvorite point-to-point chnl (PTP).

WY|X

X Y

Capacity [Shannon ’48] = suppX

I(X;Y ), and NOT (1)

Capacity [Shannon ’48] = supn≥1

suppnX

1

nI(Xn;Y n)

suppXand supn≥1 suppn

X: world of difference .

Associated with (1) is a single-letter (S-L) coding tech.

Big picture : Where is this going?

General multi-terminal problems : Seems NO single-letter (S-L) expr. forcapacity.

Example : Our fauvorite point-to-point chnl (PTP).

WY|X

X Y

Capacity [Shannon ’48] = suppX

I(X;Y ), and NOT (1)

Capacity [Shannon ’48] = supn≥1

suppnX

1

nI(Xn;Y n)

suppXand supn≥1 suppn

X: world of difference .

Associated with (1) is a single-letter (S-L) coding tech.

Big picture : Where is this going?

What if this were not true? A general multi-terminal scenario

Rx1

Rx2

Rxj

S1Tx1

Tx2

Txm

WY|X

X1

Xm

X2

Y1

Yj

Y2S2

Sm

T1

T2

Tj

Capacity > suppX|S

α(pS1···SlX1···XmY1···Ym) for every functional α(·)

Admits no optimal S-L coding techn.

This work addresses the above difficulty.

Contribution : S-L expr. for multi-letter (M-L) coding techs.

A multi-letter (M-L) coding tech. by carefully stitching together S-L techs.

Characterize (inner bound) ‘performance’ via S-L expr.Yes suppX

α(pXY ). NOT supn≥1 suppnX

.

For a general problem instance.

Derived S-L expr. strictly larger than all current known inner bnds derived viaS-L coding techs. (Examples)

Derived S-L expr. subsume current known largest inner bounds, strictly enlargefor exmpls.

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Rx1

Tx1

Tx2

S1

S2

X1

WS1S2

~~ S1

X2

Y1

2-IC

WY1Y2|X1X2

Rx2

S2Y

2

Contribution : S-L expr. for multi-letter (M-L) coding techs.

A multi-letter (M-L) coding tech. by carefully stitching together S-L techs.

Characterize (inner bound) ‘performance’ via S-L expr.Yes suppX

α(pXY ). NOT supn≥1 suppnX

.

For a general problem instance.

Derived S-L expr. strictly larger than all current known inner bnds derived viaS-L coding techs. (Examples)

Derived S-L expr. subsume current known largest inner bounds, strictly enlargefor exmpls.

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Rx1

Tx1

Tx2

S1

S2

X1

WS1S2

~~ S1

X2

Y1

2-IC

WY1Y2|X1X2

Rx2

S2Y

2

Contribution : S-L expr. for multi-letter (M-L) coding techs.

A multi-letter (M-L) coding tech. by carefully stitching together S-L techs.

Characterize (inner bound) ‘performance’ via S-L expr.Yes suppX

α(pXY ). NOT supn≥1 suppnX

.

For a general problem instance.

Derived S-L expr. strictly larger than all current known inner bnds derived viaS-L coding techs. (Examples)

Derived S-L expr. subsume current known largest inner bounds, strictly enlargefor exmpls.

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Rx1

Tx1

Tx2

S1

S2

X1

WS1S2

~~ S1

X2

Y1

2-IC

WY1Y2|X1X2

Rx2

S2Y

2

Contribution : S-L expr. for multi-letter (M-L) coding techs.

A multi-letter (M-L) coding tech. by carefully stitching together S-L techs.

Characterize (inner bound) ‘performance’ via S-L expr.Yes suppX

α(pXY ). NOT supn≥1 suppnX

.

For a general problem instance.

Derived S-L expr. strictly larger than all current known inner bnds derived viaS-L coding techs. (Examples)

Derived S-L expr. subsume current known largest inner bounds, strictly enlargefor exmpls.

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Rx1

Tx1

Tx2

S1

S2

X1

WS1S2

~~ S1

X2

Y1

2-IC

WY1Y2|X1X2

Rx2

S2Y

2

Multi-letter coding tech. : Why?? Necessary??

Prior work : Cover, El Gamal and Salehi (CES) [Nov. 1980] technique.Dueck’s example

[Nov. 1980](S1, S2) transmissible over MAC if

H(S1|S2) < I(X1;Y |X2, S2, U), H(S2|S1) < I(X2;Y |X1, S1, U)

H(S1, S2|K) < I(X1X2;Y |U), H(S1, S2) < I(X1X2;Y )

for a valid pmf WS1S2pUpX1|US1pX2|US2

WY |X1X2.

The Gacs - Korner part has to specially coded via a common conditionalcodebook.

Ignoring the GK part, treating it as soft correlation is strictly sub-optimal.

Conditional coding + separation : NO S-L long Markov chain (LMC)

S1(1) S1(2) S1(n)

X2(1) X2(2) X2(n)

U(1) U(2) U(n)

X1(1) X1(2) X1(n)

S2(1) S2(2) S2(n)

U(1) U(2) U(n)

BLOCK MAPPING

BLOCK MAPPING

X1(i) = f1i(Sn1 ) and X2(i) = f2i(S

n2 ).

X1(i)− S1(i)− S2(i)−X2(i) does NOT hold.

In fact, X1 − US1 − US2 −X2 holds. SinceU(i) = gi(K

n), we potentially have

X1(i)− Sn1 − Sn

2 −X2(i),

permitting no more reduction.

“Gacs-Korner (GK) part permitting correlation viaan n−letter LMC, and n→∞.”

Conditional coding + separation : NO S-L long Markov chain (LMC)

S1(1) S1(2) S1(n)

X2(1) X2(2) X2(n)

U(1) U(2) U(n)

X1(1) X1(2) X1(n)

S2(1) S2(2) S2(n)

U(1) U(2) U(n)

BLOCK MAPPING

BLOCK MAPPING

X1(i) = f1i(Sn1 ) and X2(i) = f2i(S

n2 ).

X1(i)− S1(i)− S2(i)−X2(i) does NOT hold.

In fact, X1 − US1 − US2 −X2 holds. SinceU(i) = gi(K

n), we potentially have

X1(i)− Sn1 − Sn

2 −X2(i),

permitting no more reduction.

“Gacs-Korner (GK) part permitting correlation viaan n−letter LMC, and n→∞.”

Conditional coding + separation : NO S-L long Markov chain (LMC)

S1(1) S1(2) S1(n)

X2(1) X2(2) X2(n)

U(1) U(2) U(n)

X1(1) X1(2) X1(n)

S2(1) S2(2) S2(n)

U(1) U(2) U(n)

BLOCK MAPPING

BLOCK MAPPING

X1(i) = f1i(Sn1 ) and X2(i) = f2i(S

n2 ).

X1(i)− S1(i)− S2(i)−X2(i) does NOT hold.

In fact, X1 − US1 − US2 −X2 holds. SinceU(i) = gi(K

n), we potentially have

X1(i)− Sn1 − Sn

2 −X2(i),

permitting no more reduction.

“Gacs-Korner (GK) part permitting correlation viaan n−letter LMC, and n→∞.”

Stripped Gacs-Korner part ⇒ Constrained to S-L LMC

S1

S2

K X1 − US1 − US2 −X2 and hence

X1 − Sn1 − Sn

2 −X2 permitted.

Stripped Gacs-Korner part ⇒ Constrained to S-L LMC

S1

S2

K1

K2

≠

P (K1 6= K2) = ξ > 0 is very tiny.

Constrained to

X1 − S1 − S2 −X2.

Discountnuous shrinkage of valid input PMFs.

Sub-optimality of Cover El Gamal Salehi rateregion.

Stripped Gacs-Korner part ⇒ Constrained to S-L LMC

S1

S2

K1

K2

≠

P (K1 6= K2) = ξ > 0 is very tiny.

Constrained to

X1 − S1 − S2 −X2.

Discountnuous shrinkage of valid input PMFs.

Sub-optimality of Cover El Gamal Salehi rateregion.

Dueck’s ingenious example [March 1981]

Rx

Tx1

Tx2

S1

S2

X1

WS1S2

~~

S1S2X2

YWY|X1X2

Identifies a sequence of MACs and source pairs WY |X1X2and WS1S2 such that

P (S1 6= S2)→ 0 and, for all source-channel pairs, sufficiently far in thissequence

For any X1 − S1 − S2 −X2, we have I(X1X2;Y ) < H(S1S2).

Yet the source pair is transmissible over the MAC.

Import of Dueck’s example

Necessary to induce correlation via a block of symbols onto channel inputs.

This necessitates a multi-letter coding.

Focus of this talk

1. Desired coding structure.

2. Overall coding tech.

3. ??challenges.??

4. Tools.

Conditional coding of GK part

⇓Correlation induced via multi-letter LMC

Absence of GK part

⇓Constrained to S-L LMC

Induce l−letter correlation via approximate conditional coding.

Approximate conditional coding of near, but not perfect, GK parts

S1n

Kn

S1n

Un

Un

Approximate conditional coding of near, but not perfect, GK parts

S1n

K1n

S1n

U1n

U2n

K2n

P (Kn1 6= Kn

2 ) = 1− (1− ξ)n → 1 as n→∞

FIX the BLOCK-LENGTH of (inner) conditional code to l.

Fixed B-L (inner) conditional code ⇒ non-vanishing Prob. of error

K1(1) K1(2) K1(l)

encoding

U1(1) U1(2) U1(l)

U l1 will be decoded with non-vanishing prob. of error.

Fixed B-L (inner) conditional code ⇒ non-vanishing Prob. of error

K1(1,1) K1(1,2) K1(1,l)

K1(m,1) K1(m,2) K1(m,l)

K1(t,1) K1(t,2) K1(t,l)

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

Row-by-row

encoding

Code over many many codewords of inner conditional code.

Number of rows m→∞.

No. of columns l remains constant.

Encoders employs identical inner codes and maps

K1(1,1) K1(1,2) K1(1,l)

K1(m,1) K1(m,2) K1(m,l)

K1(t,1) K1(t,2) K1(t,l)

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

Row-by-row

encoding

K2(1,1) K2(1,2) K2(1,l)

K2(m,1) K2(m,2) K2(m,l)

K2(t,1) K2(t,2) K2(t,l)

U2(1,1) U2(1,2) U2(1,l)

U2(m,1) U2(m,2) U2(m,l)

U2(t,1) U2(t,2) U2(t,l)

Row-by-row

encoding

Both encoders employ IDENTICAL inner codes and maps

Encoder 1

Encoder 2

Inner code error events

1. Cannot assign U−codeword to every l−length vector in K.

I τl =Prob. of unassigned U−codeword.

2. Encoders do NOT agree on a fraction of chosen U−codewords. How doyou handle this?

I Let ξ[l]:= P (Kl1 6= Kl

2) = 1− (1− ξ)l.

3. Decoder INCORRECTLY decodes a fraction of codewords on whichencoders agree on?

I Let gl denote prob of error of U−codebook on U − Y channel.

Inner code prob. of error : φ:= τl + ξ[l] + gl

How much more information needs to be sent?

Inner code decoded row by row. Let

G(1,1) G(1,2) G(1,l)

G(m,1) G(m,2) G(m,l)

G(t,1) G(t,2) G(t,l)

denote decoded version of K1,K2 matrices.

How much more information need to be sent?

Simple Fano-type upper bounding

Rows are IID. Treat as super-symbols.

G(1,1) G(1,2) G(1,l)

G(m,1) G(m,2) G(m,l)

G(t,1) G(t,2) G(t,l)

H(Sl1|Gl) ≤ H(Kl

1|Gl) +H(Sl1|Gl,Kl

1)

≤ H(Kl1,1{Kl

1 6=Gl}|Gl) +H(Sl

1|Kl1)

≤ hb(φ) + lφ log |K|+ lH(S1|K1)

≤ l(LS(φ, |K|) +H(S1|K1))

LS(φ, |K| = Loss due to incorrect conditional coding

Simple Fano-type upper bounding

Rows are IID. Treat as super-symbols.

G(1,1) G(1,2) G(1,l)

G(m,1) G(m,2) G(m,l)

G(t,1) G(t,2) G(t,l)

H(Sl1|Gl) ≤ H(Kl

1|Gl) +H(Sl1|Gl,Kl

1)

≤ H(Kl1,1{Kl

1 6=Gl}|Gl) +H(Sl

1|Kl1)

≤ hb(φ) + lφ log |K|+ lH(S1|K1)

≤ l(LS(φ, |K|) +H(S1|K1))

LS(φ, |K| = Loss due to incorrect conditional coding

Questions, Challenges

1. How do you communicate the rest of the information?

I Superposition coding.

2. How do you super-impose on codewords that induce an l−letter pmf?

I Achievable region should be in terms of pX1|U not pXl1|Ul .

Superposition coding over a matrix

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

Superposing on matrix requires optimizing over pXl1|Ul .

Desire S-L characterization. Desire suppX1|U.

Extract IID sub-vectors from this matrix and super-impose separate codebookson these sub-vectors.

Superposition coding over a matrix

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

Superposing on matrix requires optimizing over pXl1|Ul .

Desire S-L characterization. Desire suppX1|U.

Extract IID sub-vectors from this matrix and super-impose separate codebookson these sub-vectors.

The rows of Uj are independent and identically distributed.

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

U1(1,1) U1(1,2) U1(1,l)

U1(m,1) U1(m,2) U1(m,l)

U1(t,1) U1(t,2) U1(t,l)

Each row has been obtained by mapping a separate sub-block of sourcesymbols.

Interleaving [Shirani and Pradhan 2014]

Interleaving [Shirani and Pradhan 2014]

Interleaving [Shirani and Pradhan 2014]

The same argument holds for each sub-vectorsE(1, λ1(i)) · · ·E(m,λm(i)) : i = 1, · · · , l.

We therefore have l iid sub-vectors E(1, λ1(i)) · · ·E(m,λm(i)) : i = 1, · · · , l.

Randomly chosen co-ordinates are iid!!!

Co-ordinates with same color coded together within a block in the outer code.l such blocks.

Randomly chosen co-ordinates are iid!!!

Co-ordinates with same color coded together within a block in the outer code.l such blocks.

Randomly chosen co-ordinates are iid!!!

Co-ordinates with same color coded together within a block in the outer code.l such blocks.

Randomly chosen co-ordinates are iid!!!

Co-ordinates with same color coded together within a block in the outer code.l such blocks.

Csiszar’s constant composition codes

Pick test channel pU to be type of sequences of length l.

The identical U−codebook chosen at both encoders is a constant compositioncode of type pU .

Interleaved vectors will be IID pU !!!!

Upperbound the difference using relationship between chosen and actualpmf

Let pUpX1|US1pX2|US2

be chosen pmf.

pUj= pU since symbols of CU chosen iid pU .

P (U1 6= U2) ≤ β

pXj |Uj Sj= pXj |USj

: By choice of coding technique. One can derive an upper

bound using these relations.

Upperbound the difference using relationship between chosen and actualpmf

Let pUpX1|US1pX2|US2

be chosen pmf.

pUj= pU since symbols of CU chosen iid pU .

P (U1 6= U2) ≤ β

pXj |Uj Sj= pXj |USj

: By choice of coding technique. One can derive an upper

bound using these relations.

Joint source channel coding using fixed-block length

Kj(t, 1 : l) is mapped to Uj(t, 1 : l) via fixed block-length code.

Induces a correlation between Kj and Uj

Unable to characterize this correlation analytically.

Propose to destroy this correlation by coding across two m× l matrices.