sparse superposition codesquest for provably practical and reliable high rate communication the...
TRANSCRIPT
![Page 1: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/1.jpg)
SPARSE SUPERPOSITION CODES
Andrew BarronDepartment of Statistics
Yale University
Joint work with Sanghee Cho, Antony Joseph
Workshop on Modern Error Correcting CodesTokyo, Aug 30, 2013
![Page 2: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/2.jpg)
Quest for Provably Practical and ReliableHigh Rate Communication
• The Channel Communication Problem
• Shannon’s Formulation and Solution
• Gaussian Channel
• History of Methods
• Communication by Regression
• Sparse Superposition Coding
• Adaptive Successive Decoding
• Rate, Reliability, and Computational Complexity
• Distributional Analysis
![Page 3: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/3.jpg)
Shannon Formulation• Input bits: U = (U1,U2, . . . . . . ,UK ) indep Bern(1/2)
↓• Encoded: x = (x1, x2, . . . , xn)
↓• Channel: p(y |x)
↓• Received: Y = (Y1,Y2, . . . ,Yn)
↓• Decoded: U = (U1, U2, . . . . . . , UK )
• Rate: R = Kn Capacity C = maxPX I(X ; Y )
• Reliability: Want small Prob{U 6= U}and small Prob{Fraction mistakes ≥ α}
![Page 4: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/4.jpg)
Gaussian Noise Channel• Input bits: U = (U1,U2, . . . . . . ,UK )
↓• Encoded: x = (x1, x2, . . . , xn) 1
n∑n
i=1 x2i∼= P
↓• Channel: p(y |x) Y = x(U) + ε , ε ∼ N(0, σ2I)
↓ snr = P/σ2
• Received: Y = (Y1,Y2, . . . ,Yn)
↓• Decoded: U = (U1, U2, . . . . . . , UK )
• Rate: R = Kn Capacity C = 1
2 log(1 + snr)
• Reliability: Want small Prob{U 6= U}and small Prob{Fraction mistakes ≥ α}
![Page 5: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/5.jpg)
Shannon Theory
• Channel Capacity:Supremum of rates R such that reliable communication ispossible, with arbitrarily small error probability
• Information Capacity: C = maxPX I(X ; Y )
Where I(X ; Y ) is the Shannon information, also known asthe Kullback divergence between PX ,Y and PX × PY
• Shannon Channel Capacity Theorem:The supremum of achievable communication rates Requals the information capacity C
• Books:Shannon (49), Gallager (68), Cover & Thomas (06)
![Page 6: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/6.jpg)
The Gaussian Noise Model
The Gaussian noise channel is the basic model for
• wireless communicationradio, cell phones, television, satellite, space
• wired communicationinternet, telephone, cable
![Page 7: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/7.jpg)
Shannon Theory meets Coding Practice• Forney and Ungerboeck 1998 review:
• modulation, shaping and coding for the Gaussian channel• Richardson & Urbanke 2008 cover coding state of the art:
• Empirically good LDPC and turbo codes with fast encodingand decoding based on Bayesian belief networks
• Some tools for theoretical analysis, yet obstacles remain forproof of rates up to capacity for the Gaussian channel
• Arikan 2009, Arikan and Teletar 2009 polar codes:• Adapted to Gaussian channel (Abbe and Barron 2011)
• Tropp 2006,2008 codes from compressed sensing andrelated sparse signal recovery work:
• Wainwright; Fletcher, Rangan, Goyal; Zhang; others• `1-constrained least squares practical, has positive rate• but not capacity achieving
• Method here different from those above. Prior knowledgeof such is not necessary to follow what we present.
![Page 8: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/8.jpg)
Sparse Superposition Code
• Input bits: U = (U1 . . . . . . . . . . . .UK )
• Coefficients: β = (00 ∗ 0000000000 ∗ 00 . . . 0 ∗ 000000)T
• Sparsity: L entries non-zero out of N• Matrix: X , n by N, all entries indep Normal(0,1)
• Codeword: Xβ, superposition of a subset of columns• Receive: Y = Xβ + ε, a statistical linear model• Decode: β and U from X ,Y
![Page 9: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/9.jpg)
Sparse Superposition Code
• Input bits: U = (U1 . . . . . . . . . . . .UK )
• Coefficients: β = (00 ∗ 0000000000 ∗ 00 . . . 0 ∗ 000000)T
• Sparsity: L entries non-zero out of N• Matrix: X , n by N, all entries indep Normal(0,1)
• Codeword: Xβ, superposition of a subset of columns• Receive: Y = Xβ + ε, a statistical linear model• Decode: β and U from X ,Y
![Page 10: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/10.jpg)
Sparse Superposition Code
• Input bits: U = (U1 . . . . . . . . . . . .UK )
• Coefficients: β = (00 ∗ 0000000000 ∗ 00 . . . 0 ∗ 000000)T
• Sparsity: L entries non-zero out of N• Matrix: X , n by N, all entries indep Normal(0,1)
• Codeword: Xβ, superposition of a subset of columns• Receive: Y = Xβ + ε
• Decode: β and U from X ,Y• Rate: R = K
n from K = log(N
L
), near L log
(NL e)
![Page 11: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/11.jpg)
Sparse Superposition Code
• Input bits: U = (U1 . . . . . . . . . . . .UK )
• Coefficients: β = (00 ∗ 0000000000 ∗ 00 . . . 0 ∗ 000000)T
• Sparsity: L entries non-zero out of N• Matrix: X , n by N, all entries indep Normal(0,1)
• Codeword: Xβ, superposition of a subset of columns• Receive: Y = Xβ + ε
• Decode: β and U from X ,Y• Rate: R = K
n from K = log(N
L
)• Reliability: small Prob{Fraction βmistakes ≥ α}, small α
![Page 12: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/12.jpg)
Sparse Superposition Code
• Input bits: U = (U1 . . . . . . . . . . . .UK )
• Coefficients: β = (00 ∗ 0000000000 ∗ 00 . . . 0 ∗ 000000)T
• Sparsity: L entries non-zero out of N• Matrix: X , n by N, all entries indep Normal(0,1)
• Codeword: Xβ, superposition of a subset of columns• Receive: Y = Xβ + ε
• Decode: β and U from X ,Y• Rate: R = K
n from K = log(N
L
)• Reliability: small Prob{Fraction βmistakes ≥ α}, small α
Want reliability with rate up to capacity
![Page 13: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/13.jpg)
Partitioned Superposition Code• Input bits: U = (U1 . . . , . . . , . . . , . . .UK )
• Coefficients: β=(00 ∗ 00000, 00000 ∗ 00, . . . , 0 ∗ 000000)
• Sparsity: L sections, each of size M =N/L, a power of 21 non-zero entry in each section
• Indices of nonzeros: (j1, j2, . . . , jL) specified by U segments• Matrix: X , n by N, splits into L sections• Codeword: Xβ, superposition of columns, one from each• Receive: Y = Xβ + ε
• Decode: β and U• Rate: R = K
n from K = L log NL = L log M
![Page 14: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/14.jpg)
Partitioned Superposition Code• Input bits: U = (U1 . . . , . . . , . . . , . . .UK )
• Coefficients: β=(00 ∗ 00000, 00000 ∗ 00, . . . , 0 ∗ 000000)
• Sparsity: L sections, each of size M =N/L, a power of 21 non-zero entry in each section
• Indices of nonzeros: (j1, j2, . . . , jL) specified by U segments• Matrix: X , n by N, splits into L sections• Codeword: Xβ, superposition of columns, one from each• Receive: Y = Xβ + ε
• Decode: β and U• Rate: R = K
n from K = L log NL = L log M
• Reliability: small Prob{Fraction βmistakes ≥ α}, small α
Is it reliable with rate up to capacity?
![Page 15: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/15.jpg)
Partitioned Superposition Code• Input bits: U = (U1 . . . , . . . , . . . , . . .UK )
• Coefficients: β=(00 ∗ 00000, 00000 ∗ 00, . . . , 0 ∗ 000000)
• Sparsity: L sections, each of size M =N/L, a power of 21 non-zero entry in each section
• Indices of nonzeros: (j1, j2, . . . , jL) specified by U segments• Matrix: X , n by N, splits into L sections• Codeword: Xβ, superposition of columns, one from each• Receive: Y = Xβ + ε
• Decode: β and U• Rate: R = K
n from K = L log NL = L log M
• Ultra-sparse case: Impractical M = 2nR/L with L constant(reliable at all rates R < C: Cover 1972,1980)
• Moderately-sparse: Practical M = n with L = nR/ log n(Still reliable at all R < C)
![Page 16: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/16.jpg)
Partitioned Superposition Code• Input bits: U = (U1 . . . , . . . , . . . , . . .UK )
• Coefficients: β=(00 ∗ 00000, 00000 ∗ 00, . . . , 0 ∗ 000000)
• Sparsity: L sections, each of size M =N/L, a power of 21 non-zero entry in each section
• Indices of nonzeros: (j1, j2, . . . , jL) specified by U segments• Matrix: X , n by N, splits into L sections• Codeword: Xβ, superposition of columns, one from each• Receive: Y = Xβ + ε
• Decode: β and U• Rate: R = K
n from K = L log NL = L log M
• Reliability: small Prob{Fraction mistakes ≥ α} small α• Outer RS code: rate 1−α, corrects remaining mistakes• Overall rate: Rtot = (1−α)R• Overall rate: up to capacity
![Page 17: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/17.jpg)
Power Allocation
• Coefficients: β=(00∗00000, 00000∗00, . . . ,0∗000000)
• Indices of nonzeros: sent = (j1, j2, . . . , jL)
• Coeff. values: βj` =√
P` for ` = 1,2, . . . ,L
• Power control:∑L
`=1 P` = P
• Codewords: Xβ, have average power P
• Power Allocations
• Constant power: P` = P/L
• Variable power: P` proportional to e−2C `/L
![Page 18: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/18.jpg)
Variable Power Allocation
• Power control:∑L
`=1 P` = P ‖β‖2 = P
• Variable power: P` proportional to e−2C`/L for ` = 1, . . . ,L
0 20 40 60 80 100
0.000
0.005
0.010
0.015
0.020
section index
pow
er a
lloca
tion
![Page 19: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/19.jpg)
Variable Power Allocation
• Power control:∑L
`=1 P` = P ‖β‖2 = P
• Variable power: P` proportional to e−2C`/L for ` = 1, . . . ,L
• Successive decoding motivation• Incremental capacity
12
log(
1 +P`
σ2 + P`+1 + · · ·+ PL
)=
CL
matching the section rate
RL
=log M
n
![Page 20: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/20.jpg)
Adaptive Successive Decoder
Decoding Steps (with thresholding)
• Start: [Step 1]• Compute the inner product of Y with each column of X• See which are above a threshold• Form initial fit as weighted sum of columns above threshold
• Iterate: [Step k ≥ 2]• Compute the inner product of residuals Y − Fitk−1 with
each remaining column of X• Standardize by dividing by ‖Y − Fitk−1‖• See which are above the threshold
√2 log M + a
• Add these columns to the fit
• Stop:• At Step k = 1 + snr log M, or• if there are no additional inner products above threshold
![Page 21: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/21.jpg)
Complexity of Adaptive Successive Decoder
Complexity in parallel pipelined implementation
• Space: (use k = snr log M copies of the n by N dictionary)
• knN = snr C M n2 memory positions• kN multiplier/accumulators and comparators
• Time: O(1) per received Y symbol
![Page 22: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/22.jpg)
Adaptive Successive DecoderDecoding Steps (with iteratively optimal statistics)
• Start: [Step 1]• Compute the inner product of Y with each column of X• Form initial fit
• Iterate: [Step k ≥ 2]• Compute inner product of residuals Y − Fitk−1 with each Xj .• Adjusted form: statk,j equals (Y − X βk−1,−j )
T Xj
• Standardize by dividing it by ‖Y − X βk−1‖.• Form the new fit
βk,j =√
P` wj (α) =√
P`eα statk,j∑
j∈sec` eα statk,j
• Stop:• At Step k = O(log M), sufficient if R < C.
![Page 23: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/23.jpg)
Iteratively Bayes optimal βkWith prior j`∼Unif on sec`, the Bayes estimate based on statk
βk = E[β|statk ]
has representation βk ,j =√
P` wk ,j with
wk ,j = Prob{j` = j |statk}.
Here, when the statk ,j are independent N(α`,k1{j=j`},1), wehave the logit representation wk ,j = w(α`,k ) where
wk ,j(α) =eα statk,j∑
j∈sec` eα statk,j.
![Page 24: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/24.jpg)
Iteratively Bayes optimal βkWith prior j`∼Unif on sec`, the Bayes estimate based on statk
βk = E[β|statk ]
has representation βk ,j =√
P` wk ,j with
wk ,j = Prob{j` = j |statk}.
Here, when the statk ,j are independent N(α`,k1{j=j`},1), wehave the logit representation wk ,j = w(α`,k ) where
wk ,j(α) =eα statk,j∑
j∈sec` eα statk,j.
![Page 25: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/25.jpg)
Iteratively Bayes optimal βkWith prior j`∼Unif on sec`, the Bayes estimate based on statk
βk = E[β|statk ]
has representation βk ,j =√
P` wk ,j with
wk ,j = Prob{j` = j |statk}.
Here, when the statk ,j are independent N(α`,k1{j=j`},1), wehave the logit representation wk ,j = w(α`,k ) where
wk ,j(α) =eα statk,j∑
j∈sec` eα statk,j.
Recall statk ,j is standardized inner product of residuals with Xj
statk ,j =(Y − X βk−1,−j)
T Xj
‖Y − X βk−1‖
![Page 26: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/26.jpg)
Distributional Analysis• Approximate distribution of statk .
statk ,j =
√n βj√
σ2 + E‖βk−1 − β‖2+ Zk ,j
independent standard normal, shifted for terms sent
statk ,j = α`,xk−11{j sent} + Zk ,j
where
α = α`,x =
√nP`
σ2 + P(1− x)
• HereE‖βk−1 − β‖2 = P (1− xk−1)
• Update rule xk = g(xk−1) where
g(x) =L∑`=1
(P`/P)E [wj`(α`,x )].
![Page 27: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/27.jpg)
Distributional Analysis• Approximate distribution of statk .
statk ,j =
√n βj√
σ2 + E‖βk−1 − β‖2+ Zk ,j
independent standard normal, shifted for terms sent
statk ,j = α`,xk−11{j sent} + Zk ,j
where
α = α`,x =
√nP`
σ2 + P(1− x)
• HereE‖βk−1 − β‖2 = P (1− xk−1)
• Update rule xk = g(xk−1) where
g(x) =L∑`=1
(P`/P)E [wj`(α`,x )].
![Page 28: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/28.jpg)
Success Rate Update Rule
• Update rule xk = g(xk−1) where
g(x) =L∑`=1
(P`/P)E [wj`(α`,x )]
• this is the success rate update function expressed as aweighted sum of the posterior prob of the term sent
![Page 29: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/29.jpg)
Decoding progression
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x
M = 29 , L =Msnr=7C=1.5 bitsR=1.2 bits(0.8C)
g(x)x
Figure : Plot of g(x) and the sequence xk .
![Page 30: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/30.jpg)
Update fuctions
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x
M = 29 , L =Msnr=7C=1.5 bitsR=1.2 bits(0.8C)
g(x)Lower bounda=0a=0.5
Figure : Comparison of update functions. Blue and light blue linesindicates {0,1} decision using the threshold τ =
√2logM + a with
respect to the value a as indicated.
![Page 31: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/31.jpg)
Transition plots0.0
0.4
0.8
ul
ew1
post
erio
r pro
b of
the
term
sen
t
x=0
soft decisionhard decision with a=1/2
ulew1
x=0.2
ul
ew1
x=0.4
0.0 0.2 0.4 0.6 0.8
0.0
0.4
0.8
ul
ew1
u(l)
x=0.6
0.0 0.2 0.4 0.6 0.8
ul
ew1
u(l)
x=0.8
0.0 0.2 0.4 0.6 0.8
ul
ew1
u(l)
x=1
Figure : Transition plots : M = 29, L = M, C = 1.5 bits and R = 0.8C.We used Monte Carlo simulation with replicate size 10000. Thehorizontal axis depicts u(`) = 1− e−2C`/L which is an increasingfunction of `. Area under curve equals g(x).
![Page 32: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/32.jpg)
Rate and Reliability
Result for Optimal ML Decoder [Joseph and B. 2012],with outer RS decoder, and with equal power allowed acrossthe sections
• Prob error exponentially small in n for all R < C
Prob{Error} ≤ e−n (C−R)2/2V
• In agreement with the Shannon-Gallager exponent ofoptimal code, though with a suboptimal constant Vdepending on the snr
![Page 33: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/33.jpg)
Rate and Reliability of Fast Superposition Decoder
Practical: Adaptive Successive Decoder, with outer RS code.
• prob error exponentially small in n/(log M) for R < C
• Value CM approaching capacity
CM =C
1 + c1/ log M
• Probability error exponentially small in L for R < CM
Prob{
Error}≤ e−L(CM−R)2c2
• Improves to e−c3L(CM−R)2(log M)0.5using a Bernstein bound.
![Page 34: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/34.jpg)
Summary
Sparse superposition codes with adaptive successive decoding
• Simplicity of the code permits:• distributional analysis of the decoding progression• low complexity decoder• exponentially small error probability for any fixed R < C
• Asymptotics superior to polar code bounds for such rates
![Page 35: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/35.jpg)
Sparse Superposition Code for the Gaussian Channel
uInput bits(length K )
βSparse
coeff. vector(length N)L non-zero‖β‖2 = P
XDictionary
n by Nindep N(0,1)
XβCodeword(length n)
Channel
εNoise ∼ N(0, σ2I)
Yreceived(length n)
Decoder u
![Page 36: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/36.jpg)
Sparse Superposition Code for the Gaussian Channel
uInput bits(length K )
βSparse
coeff. vector(length N)L non-zero‖β‖2 = P
XDictionary
n by Nindep N(0,1)
XβCodeword(length n)
Channel
εNoise ∼ N(0, σ2I)
Yreceived(length n)
Decoder u
Linear Model Y = Xβ + ε
![Page 37: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/37.jpg)
Sparse Superposition Code for the Gaussian Channel
uInput bits(length K )
βSparse
coeff. vector(length N)L non-zero‖β‖2 = P
XDictionary
n by Nindep N(0,1)
XβCodeword(length n)
Channel
εNoise ∼ N(0, σ2I)
Yreceived(length n)
Decoder u
• Partitioned Coef.: β = (00∗0000, 000∗000, . . . ,0∗00000)
• L sections of size M = N/L, one non-zero in each
• Rate R = Kn = L log M
n , Capacity C = 12 log(1 + snr)
Maximum likelihood decoder (Joseph & Barron 2010a ISIT, 2012a IT)
Adaptive successive decoder with threshold (J&B 2010b ISIT, 2012b)
Adaptive successive decoder with soft decision (B&C ISIT 2012)
![Page 38: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/38.jpg)
Sparse Superposition Code for the Gaussian Channel
uInput bits(length K )
βSparse
coeff. vector(length N)L non-zero‖β‖2 = P
XDictionary
n by Nindep N(0,1)
XβCodeword(length n)
Channel
εNoise ∼ N(0, σ2I)
Yreceived(length n)
Decoder u
snr = Pσ2
• Partitioned Coef.: β = (00∗0000, 000∗000, . . . ,0∗00000)
• L sections of size M = N/L, one non-zero in each
• Rate R = Kn = L log M
n , Capacity C = 12 log(1 + snr)
Maximum likelihood decoder (Joseph & Barron 2010a ISIT, 2012a IT)
Adaptive successive decoder with threshold (J&B 2010b ISIT, 2012b)
Adaptive successive decoder with soft decision (B&C ISIT 2012)
![Page 39: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/39.jpg)
Sparse Superposition Code for the Gaussian Channel
uInput bits(length K )
βSparse
coeff. vector(length N)L non-zero‖β‖2 = P
XDictionary
n by Nindep N(0,1)
XβCodeword(length n)
Channel
εNoise ∼ N(0, σ2I)
Yreceived(length n)
Decoder u
snr = Pσ2
• Partitioned Coef.: β = (00∗0000, 000∗000, . . . ,0∗00000)
• L sections of size M = N/L, one non-zero in each
• Rate R = Kn = L log M
n , Capacity C = 12 log(1 + snr)
Maximum likelihood decoder (Joseph & Barron 2010a ISIT, 2012a IT)
Adaptive successive decoder with threshold (J&B 2010b ISIT, 2012b)
Adaptive successive decoder with soft decision (B&C ISIT 2012)
![Page 40: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/40.jpg)
Iterative Estimation
For k ≥ 1
• Codeword fits: Fk = X βk
• Vector of statistics: statk = function of (X ,Y ,F1, . . . ,Fk )
• e.g. statk ,j proportional to X Tj (Y − Fk )
• Update βk+1 as a function of statk
• Thresholding: Adaptive Successive Decoderβk+1,j =
√P` 1{statk,j>thres}
• Soft decision:βk+1,j = E[βj |statk ] =
√P` wk,j
with thresholding on the last step
![Page 41: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/41.jpg)
Iterative Estimation
For k ≥ 1
• Codeword fits: Fk = X βk
• Vector of statistics: statk = function of (X ,Y ,F1, . . . ,Fk )
• e.g. statk ,j proportional to X Tj (Y − Fk )
• Update βk+1 as a function of statk
• Thresholding: Adaptive Successive Decoderβk+1,j =
√P` 1{statk,j>thres}
• Soft decision:βk+1,j = E[βj |statk ] =
√P` wk,j
with thresholding on the last step
![Page 42: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/42.jpg)
Statistics
• Codeword fits: Fk = X βk
• Orthogonalization : Let G0 = Y and for k ≥ 1
Gk = part of Fk orthogonal to G0,G1, . . . ,Gk−1
• Components of statistics
Zk ,j =X T
j Gk
‖Gk‖
• Statistics such as statk built from X Tj (Y − Fk ,−j) are linear
combinations of these Zk ,j
![Page 43: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/43.jpg)
Distributional AnalysisLemma 1: shifted normal conditional distributionGiven Fk−1 = (‖G0‖, . . . , ‖Gk−1‖,Z0,Z1, . . . ,Zk−1), the Zk hasthe distributional representation
Zk =‖Gk‖σk
bk + Zk
• ‖Gk‖2/σ2k ∼ Chi-square(n − k)
• b0,b1, . . . ,bk the successive orthonormal components of[βσ
],
[β10
], . . . ,
[βk0
](∗)
• Zk ∼ N(0,Σk ) indep of ‖Gk‖
• Σk = I − b0bT0 − b1bT
1 − . . .− bkbTk
= projection onto space orthogonal to (∗)
• σ2k = βT
k Σk−1βk
![Page 44: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/44.jpg)
Idealized statisticsClass of statistics statk formed by combining Z0, . . . ,Zk
statk ,j = Zcombk ,j +
√n√ckβk ,j
with Zcombk = λ0,k Z0 +λ1,k Z1 + . . .+λk ,k Zk ,
∑kk ′=0 λ
2k ′,k = 1
Desired form of the statistics
stat idealk =
√n√ckβ + Z comb
k
for ck = σ2 + (1− xk )P
For the terms sent the shift α`,k has an effective snrinterpretation
α`,k =
√n
P`ck
=
√n
P`σ2 + Premaining,k
![Page 45: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/45.jpg)
Idealized statisticsClass of statistics statk formed by combining Z0, . . . ,Zk
statk ,j = Zcombk ,j +
√n√ckβk ,j
with Zcombk = λ0,k Z0 +λ1,k Z1 + . . .+λk ,k Zk ,
∑kk ′=0 λ
2k ′,k = 1
Desired form of the statistics
stat idealk =
√n√ckβ + Z comb
k
for ck = σ2 + (1− xk )P
For the terms sent the shift α`,k has an effective snrinterpretation
α`,k =
√n
P`ck
=
√n
P`σ2 + Premaining,k
![Page 46: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/46.jpg)
Example of statisticsWeights of combination: based on λk proportional to(
(σY − bT0 βk ), −(bT
1 βk ), . . . , −(bTk βk )
)If we combine Zk , replacing χn−k with
√n, it produces the
desired distributional representation
statk =
√n√
σ2 + ‖β − βk‖2β + Z comb
k
with Z combk ∼ N(0, I) and σ2
Y = σ2 + P.
• We cannot calculate the weights not knowing βe.g. b0 = β/σ
• ‖β − βk‖2 is close to its known expectation• This provides approximation of the distribution of the statk ,j
as independent shifted normals.
![Page 47: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/47.jpg)
Example of statisticsWeights of combination: based on λk proportional to(
(σY − bT0 βk ), −(bT
1 βk ), . . . , −(bTk βk )
)If we combine Zk , replacing χn−k with
√n, it produces the
desired distributional representation
statk =
√n√
σ2 + ‖β − βk‖2β + Z comb
k
with Z combk ∼ N(0, I) and σ2
Y = σ2 + P.• We cannot calculate the weights not knowing β
e.g. b0 = β/σ
• ‖β − βk‖2 is close to its known expectation• This provides approximation of the distribution of the statk ,j
as independent shifted normals.
![Page 48: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/48.jpg)
Example of statisticsWeights of combination: based on λk proportional to(
(σY − bT0 βk ), −(bT
1 βk ), . . . , −(bTk βk )
)Estimated weights of combination proportional to(
(‖Y‖ − ZT0 βk ), −(ZT
1 βk ), . . . , −(ZTk βk )
)These estimated weights produce the residual based statisticspreviously discussed
![Page 49: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/49.jpg)
Using idealized weights of combinationsFor given β1, . . . , βk and c0 > . . . > ck , ck = σ2 + (1− xk )PWe combine Zk ′ =
√n bk ′ + Zk ′ with
λ∗k =√
ck
(√1c0,−
√1c1− 1
c0, . . . ,−
√1ck− 1
ck−1
)Approximately optimal estimate is based on
statk =k∑
k ′=0
λ∗k ′,kZk ′ +
√n√ckβk
Lemma: For 1 ≤ k ≤ k∗, for any δ > 0,
|βT βk − xkP| ≤ ak (n/L)k−1/2δ n/L = (log M)/R
except an event of probability of
k∗exp{−Lδ2/16}+ 6k∗exp{− 2c2 Lδ2}
![Page 50: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/50.jpg)
Cholesky decomposition based estimates
Recall if we combine√
n bk + Zk with weights λ∗k proportional to(
(σY − bT0 βk ), −(bT
1 βk ), . . . , −(bTk βk )
)producing the desired distributional representation
ˆstat∗k =
√n√
σ2 + ‖β − βk‖2β + Z comb
k
we cannot calculate λ∗k not knowing β
![Page 51: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/51.jpg)
Estimate weight of combinationEstimation of(
(σY − bT0 βk ), −(bT
1 βk ), . . . , −(bTk βk )
)These bT
k ′ βk arise inthe QR-decomposition for B = [β, β1, . . . , βk ]
![Page 52: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/52.jpg)
Estimate weight of combination
These bTk ′ βk arise in the Cholesky decomposition BT B =RT R,
The ZTk βk/
√n provides estimates for the diagonals of R using
ZTk βk/
√n = (bT
k βk )χn−k/√
n
The error of this estimate is controlled by χn−k/√
n − 1.Based on the estimates we can recover the rest of thecomponents of the matrix.
![Page 53: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/53.jpg)
Estimate weights of combinations
Lemma: For 1 ≤ k ′ < k ≤ k∗,
|bTk ′ βk − rk ′,k | ≤ FR,k ′,k
√L/n δ
except an event of probability of 6k∗e−Lδ2.
• FR,k ′,k is a function of elements from the upper k ′ × k partof the matrix R.
• We believe each bTk ′ βk is close to deterministic value.
• If it is true, FR,k ′,k is close to FR∗,k ′,k where R∗ is replacingbT
k ′ βk with deterministic constants.
![Page 54: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/54.jpg)
Estimate weights of combinations
Lemma: For 1 ≤ k ′ < k ≤ k∗,
|bTk ′ βk − rk ′,k | ≤ FR,k ′,k
√L/n δ
except an event of probability of 6k∗e−Lδ2.
• FR,k ′,k is a function of elements from the upper k ′ × k partof the matrix R.
• We believe each bTk ′ βk is close to deterministic value.
• If it is true, FR,k ′,k is close to FR∗,k ′,k where R∗ is replacingbT
k ′ βk with deterministic constants.
![Page 55: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/55.jpg)
Strategy
Current investigation, show inductively, for k ≥ 1,
(a)∣∣∣βT βk − xkP
∣∣∣ ≤√n/L δ
(b)∣∣∣‖βk‖2 − xkP
∣∣∣ ≤√n/L δ
(c)∣∣∣βT
k ′ βk − xk ′P∣∣∣ ≤√n/L δ for k ′ = 1, . . . , k − 1.
(d)∣∣∣bT
k ′ βk − ck
√1
ck′− 1
ck′−1
∣∣∣ ≤√n/L δ for k ′ = 1, . . . , k − 1.
(e)∣∣∣bT
k ′ βk − rk ′,k
∣∣∣ ≤√L/n δ for k ′ = 1, . . . , k − 1.
![Page 56: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/56.jpg)
Conclusion
• Iterative estimation for sparse superposition codes foradditive white Gaussian noise channel.
• Iteratively obtained statistics Zk ,j are approximately shiftednormal distributed.
• The update function shows the progression of theestimation approximately
• There are different choices of function of statistics statk weuse for the estimation
• Using residuals• Using idealized weights of combinations• Cholesky decomposition based estimates
Thank you!
![Page 57: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/57.jpg)
Rate versus Section Size
0.0
0.5
1.0
1.5
2.0
B
R (
bits
/ch.
use
)
●
● ●●
26 28 210 212 214 216
●
detailed envelopecurve when L == Bcurve using simulation runscapacitysnr == 15
Figure : Rate as a function of B for snr =15 and error probability10−3.
![Page 58: SPARSE SUPERPOSITION CODESQuest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Shannon’s Formulation and Solution Gaussian Channel](https://reader034.vdocuments.us/reader034/viewer/2022042122/5e9ca64b85a7ae345b2c2615/html5/thumbnails/58.jpg)
Rate versus Section Size
0.0
0.1
0.2
0.3
0.4
0.5
B
R (
bits
/ch.
use
)
●
●
●
26 28 210 212 214 216
●
detailed envelopecurve when L == Bcurve using simulation runscapacitysnr == 1
Figure : Rate as a function of B for snr = 1 and error probability 10−3.