capacity of noiseless and noisy two-dimensional...
TRANSCRIPT
![Page 1: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/1.jpg)
1/11/05 1
Capacity of Noiseless and Noisy Two-Dimensional Channels
Paul H. Siegel
Electrical and Computer Engineering Center for Magnetic Recording Research
University of California, San Diego
![Page 2: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/2.jpg)
LANL Workshop 1/11/05 2
Outline
• Shannon Capacity • Discrete-Noiseless Channels
• One-dimensional • Two-dimensional
• Finite-State Noisy Channel • One-dimensional • Two-dimensional
• Summary
![Page 3: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/3.jpg)
LANL Workshop 1/11/05 3
Claude E. Shannon
![Page 4: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/4.jpg)
LANL Workshop 1/11/05 4
The Inscription
![Page 5: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/5.jpg)
LANL Workshop 1/11/05 5
The Formula on the “Paper”
Capacity of a discrete channel with noise [Shannon, 1948]
For noiseless channel, Hy(x)=0, so:
Gaylord, MI: C = W log (P+N)/N Bell Labs: no formula on paper (“H = – p log p – q log q” on plaque)
(x))H – (H(x)Max C y=
H(x)Max C =
![Page 6: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/6.jpg)
LANL Workshop 1/11/05 6
Discrete Noiseless Channels (Constrained Systems)
• A constrained system S is the set of sequences generated by walks on a labeled, directed graph G.
Telegraph channel constraints [Shannon, 1948]
DOT
DASH
DOT
DASH
LETTER SPACE
WORD SPACE
![Page 7: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/7.jpg)
LANL Workshop 1/11/05 7
Magnetic Recording Constraints
Forbidden words F={101, 010}
0 1
0 0
1 1 Biphase
Even
1 0
0 0
1 1
1
0
1
0
Forbidden word F={11}
Runlength constraints (“finite-type”: determined by finite
list F of forbidden words)
Spectral null constraints (“almost-finite-type”)
![Page 8: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/8.jpg)
LANL Workshop 1/11/05 8
(d,k) runlength-limited constraints
• For , a (d,k) runlength-limited sequence is a binary string such that:
• F={11} forbidden list corresponds to
1 0 0 0 1 0 1 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 0
<≤ kd0
),1(),( ∞=kd
sd consecbetween '0# ≤
![Page 9: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/9.jpg)
LANL Workshop 1/11/05 9
Practical Constrained Codes
Finite-state encoder Sliding-block decoder (from binary data into S) (inverse mapping from S to data)
m data bits n code bits
Encoder Logic
Rate m:n
(states)
Decoder Logic
n bits
m bits We want: high rate R=m/n low complexity
![Page 10: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/10.jpg)
LANL Workshop 1/11/05 10
Codes and Capacity
• How high can the code rate be? • Shannon defined the capacity of the constrained system S:
where N(S,n) is the number of sequences in S of length n.
Theorem [Shannon,1948] : If there exists a decodable code at rate R= m/n from binary data to S, then R W C.
Theorem [Shannon,1948] : For any rate R=m/n < C there exists a block code from binary data to S with rate km:kn, for some integer k D 1.
( )S,n Nn
Cn
log1lim
∞→
=
![Page 11: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/11.jpg)
LANL Workshop 1/11/05 11
Computing Capacity: Adjacency Matrices
• Let be the adjacency matrix of the graph G representing S.
• The entries in correspond to paths in G of length n.
GA
nGA
0
0
1
=
0111
GA
![Page 12: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/12.jpg)
LANL Workshop 1/11/05 12
Computing Capacity (cont.)
• Shannon showed that, for suitable representing graphs G , where , i.e., the spectral radius of the matrix .
• Assigning “transition probabilities” to the edges of G, the constrained system S becomes a Markov source x, with entropy H(x). Shannon proved that
and expressed the maximizing probabilities in terms of the spectral radius and corresponding eigenvector of .
( )GA ρlogC =
( ) { }GG lue of Aan eigenva is: λ λ maxAρ =GA
( )x HmaxC =
GA
![Page 13: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/13.jpg)
LANL Workshop 1/11/05 13
Maxentropic Measure
• Let denote the largest real eigenvalue of , with
corresponding eigenvector • Then the maxentropic (capacity-achieving) transition
probabilities are given by
• The stationary state distribution is expressed in terms of corresponding left and right eigenvectors.
λ[ ]MBBB ,,1 =
λij
i
jij
ABB
P ⋅=
GA
![Page 14: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/14.jpg)
LANL Workshop 1/11/05 14
Computing Capacity (cont.)
• Example:
• More generally, , where is the largest real root of the polynomial and
),1(),( ∞=kd
6942.02
51log ≈+
=C
kdkd logC ,, λ= kd ,λ
∞<−−−−= −+ kxxxxf dkkkd for ,1)( 1
,
. 1for ,12 ,1, ≥= −−∞ dCC ddd
![Page 15: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/15.jpg)
LANL Workshop 1/11/05 15
Constrained Coding Theorems
• Stronger coding theorems were motivated by the problem of constrained code design for magnetic recording.
Theorem[Adler-Coppersmith-Hassner, 1983] Let S be a finite-type constrained system. If m/n ≤ C, then
there exists a rate m:n sliding-block decodable, finite-state encoder.
(Proof is constructive: state-splitting algorithm.) Theorem[Karabed-Marcus, 1988] Ditto if S is almost-finite-type. (Proof not so constructive…)
![Page 16: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/16.jpg)
LANL Workshop 1/11/05 16
Two-Dimensional Constrained Systems
• Band-recording and page-oriented recording technologies require 2-dimensional constraints, for example:
• Two-Dimensional Optical Storage (TwoDOS) - Philips • Holographic Storage - InPhaseTechnologies • Patterned Magnetic Media – Hitachi, Toshiba, … • Thermo-Mechanical Probe Array – IBM
![Page 17: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/17.jpg)
LANL Workshop 1/11/05 17
TwoDOS
Courtesy of Wim Coene, Philips Research
![Page 18: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/18.jpg)
LANL Workshop 1/11/05 18
Constraints on the Integer Lattice Z2
• : constraint in x - y directions:
),1(),( ∞=kd
Hard-Square Model
1 1
1
1
1
1
1 1
1
1
1 1
1 1
1
1 Independent Sets
[ ]
= 11,
11
F
∞,1sqS
![Page 19: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/19.jpg)
LANL Workshop 1/11/05 19
(d,k) Constraints on the Integer Lattice Z2
• For 2-dimensional (d,k) constraints , the capacity is given by:
• The only nontrivial (d,k) pairs for which is known precisely are those with zero capacity, namely [Kato-Zeger, 1999] :
mnN
limCkdnm
nm
kd,,
,
,
∞→=
kdC ,
2 , 0, +≥> dkC kd
kdsqS ,
01, =+ddC , d>0
![Page 20: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/20.jpg)
LANL Workshop 1/11/05 20
(d,k) Constraints on Z2 – Capacity Bounds
• Transfer matrix methods provide numerical bounds on [Calkin-Wilf, 1998] , [Nagy-Zeger, 2000]
• Variable-rate “bit-stuffing” encoders for yield best known lower bounds on for d >1 [Halevy, et al., 2004]:
∞,dC
6858789116180 7558789116170 ,1 .. C ≤≤ ∞
∞,dsqS
∞,1C
d Lower bound d Lower bound
2 0.4267 4 0.2858 3 0.3402 5 0.2464
( ) ( ) )1(121)(max lim /},min{12210,
,dnmdpnm
d oppdp
phC −−−+
≥ −<<∞→
∞
![Page 21: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/21.jpg)
LANL Workshop 1/11/05 21
2-D Bit-Stuffing RLL Encoder
• Source encoder converts binary data to i.i.d bit stream
(biased bits) with , rate penalty . • Bit-stuffing encoder inserts redundant bits which can be
identified uniquely by decoder. • Encoder rate R(p) is a lower bound of the capacity.
(For d=1, we can determine R(p) precisely.)
Data Source
Bit-stuffingDecoder
SourceEncoder
Bit-stuffingEncoder
DiscreteNoiselessChannel
SourceDecoderData Sink
( ) ( ) pp, −== 10Pr 1Pr )( ph
),( ∞d
![Page 22: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/22.jpg)
LANL Workshop 1/11/05 22
2-D Bit-Stuffing (1,∞) RLL Encoder
• Biased sequence: 1 1 1 0 0 0 1 0 0 1 0 0 0 0 1 1 0 0 0
1 0
0
0
0
0 0 1
1
1
1
1
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
0 0
0
0
Optimal bias Pr(1) = p = 0.3556 R(p)=0.583056 (within 1% of capacity)
![Page 23: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/23.jpg)
LANL Workshop 1/11/05 23
Enhanced Bit-Stuffing Encoder
• Use 2 source encoders, with parameters p0 , p1 .
0
0
0 0
0
1
Optimal bias
Pr(1) = p0 = 0.328167
Optimal bias
Pr(1) = p1 = 0.433068
R(p0 , p1)=0.587277 (within 0.1% of capacity)
![Page 24: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/24.jpg)
LANL Workshop 1/11/05 24
Non-Isolated Bit (n.i.b.) Constraint on Z2
• The non-isolated bit constraint is defined by the forbidden set:
• Analysis of the coding ratio of a bit-stuffing encoder yields:
0.91276 ≤ Csqnib ≤ 0.93965
=
1101
1 ,
0010
0 F
nibsqS
![Page 25: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/25.jpg)
LANL Workshop 1/11/05 25
Constraints on the Hexagonal Lattice A2
• : constraints:
),1(),( ∞=kd
Hard-Hexagon Model
∞,1hexS
[ ]
=
11
,11,11
F
⇔ 1 1 1
1
1 1
1
1
![Page 26: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/26.jpg)
LANL Workshop 1/11/05 26
Hard Hexagon Capacity
• Capacity of hard hexagon model is known precisely! [Baxter,1980]*
212
4
22
3
22
2
21254511
1221
12211
12211
1134
−
−−−
+++++−=
+++++−−−=
+++++−−=
=
aaaa
cccc
cccc
c
κ
κ
κ
κ
∞,1hexC
[ ]31
3131
21
31
)1()1(83
41
33119792501
11363124
−−++=
=
−=
bbac
b
a
and where,log 4321,1 κκκκκκ ==∞
hhexC
So, 480767622.0,1 ≈∞hexC
![Page 27: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/27.jpg)
LANL Workshop 1/11/05 27
Hard Hexagon Capacity
• Alternatively, the hard hexagon entropy constant satisfies a degree-24 polynomial with (big!) integer coefficients.
• Baxter does offer this disclaimer regarding his derivation, however:
κ
*“It is not mathematically rigorous, in that certain analyticity properties of κ are assumed, and the results of Chapter 13 (which depend on assuming that various large-lattice limits can be interchanged) are used. However, I believe that these assumptions, and therefore (14.1.18)-(14.1.24), are in fact correct.”
![Page 28: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/28.jpg)
LANL Workshop 1/11/05 28
(d,k) Constraints on A2 – Capacity Bounds
• Zero capacity region partially known [Kukorelly-Zeger, 2001]. • Variable-to-fixed length “bit-stuffing” encoders for yield best known lower bounds on for d>1
∞,dhexS
∞,dhexC
d Lower bound d Lower bound
2 0.3387 4 0.2196
3 0.2630 5 0.1901
[Halevy, et al., 2004]:
( ) )1(31
)(max lim /},min{210,
,dnmpnm
dhex o
pdpphC −−+
≥<<∞→
∞
![Page 29: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/29.jpg)
LANL Workshop 1/11/05 29
Practical 2-D Constrained Codes
• There is no comprehensive algorithmic theory for constructing encoders and decoders for 2-D constrained systems.
• Very efficient bit-stuffing encoders have been defined and analyzed for several 2-D constraints, but they are not suitable for practical applications [Roth et al., 2001] , [Halevy et al., 2004] , [Nagy-Zeger, 2004].
• Optimal block codes with m x n rectangular code arrays have been designed for small values of m and n, and some finite-state encoders have been designed, but there is no generally applicable method [Demirkan-Wolf, 2004] .
![Page 30: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/30.jpg)
LANL Workshop 1/11/05 30
Concluding Remarks
• The lack of convenient graph-based representations of 2-D constraints prevents the straightforward extension of 1-D techniques for analysis and code design.
• There are strong connections to statistical physics that may open up new approaches to understanding 2-D constrained systems (and, perhaps, vice-versa).
![Page 31: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/31.jpg)
LANL Workshop 1/11/05 31
Noisy Finite-State ISI Channels (1-Dim.)
• Binary input process
• Linear intersymbol interference
• Additive, i.i.d. Gaussian noise
∑−
=
+−=1
0][][][][
n
kinkixkhiy
][ix
][ih
)( 2,0~][ σNin
![Page 32: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/32.jpg)
LANL Workshop 1/11/05 32
Example: Partial-Response Channels
• Impulse response:
• Example: Dicode channel
1
0)1)(1(][)( −
=
+−==∑ NiN
iDDDihDh
)1()( DDh −=
-1
1 0 0
0 1
![Page 33: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/33.jpg)
LANL Workshop 1/11/05 33
Entropy Rates
• Output entropy rate:
• Noise entropy rate:
• Conditional entropy rate:
( ) ( )nn YH
nYH 1
1lim ∞→=
( ) ( )021 eNlogNH π=
( ) ( ) ( )NHXYHn
XYH nn
n==
∞→ 11 |1lim|
![Page 34: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/34.jpg)
LANL Workshop 1/11/05 34
Mutual Information Rates
• Mutual information rate:
• Capacity:
• Symmetric information rate (SIR): Inputs are constrained to be independent, identically distributed, and equiprobable binary digits.
( ) ( ) ( ) ( ) ( )NHYHX|YHYHY;XI −=−=
{ }][ixX =
( )( )Y;XImaxC
XP=
![Page 35: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/35.jpg)
LANL Workshop 1/11/05 35
Finding the Output Entropy Rate
• For one-dimensional ISI channel model:
and where
( ) ( )n
nYH
nYH 1
1lim∞→
=
( ) ( )[ ]nnn yYpEYH 111 log =−=
[ ] [ ] [ ][ ]nYYYY n ,2,11 =
![Page 36: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/36.jpg)
LANL Workshop 1/11/05 36
Sample Entropy Rate
• If we simulate the channel N times, using inputs with specified (Markovian) statistics and generating output realizations
then
converges to with probability 1 as .
∞→N( )nYH 1
[ ] Nknyyyy kkkk ,,2,1,][,,]2[,]1[ )()()()( ==
( )∑=
−N
k
kypN 1
)(log1
![Page 37: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/37.jpg)
LANL Workshop 1/11/05 37
Computing Sample Entropy Rate • The forward recursion of the sum-product (BCJR) algorithm can be used to calculate the probability p(y1
n) of a sample realization of the channel output. • In fact, we can write
where the quantity is precisely the normalization constant in the (normalized) forward recursion.
( ) ( )∑=
−−=−n
i
ii
n y|yplogn
yplogn 1
111
11
( )11−i
i y|yp
![Page 38: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/38.jpg)
LANL Workshop 1/11/05 38
Computing Entropy Rates
• Shannon-McMillan-Breimann theorem implies
as , where is a single long sample
realization of the channel output process.
ny1
( ) ( )YHyplogn .s.a
n →− 11
∞→n
![Page 39: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/39.jpg)
LANL Workshop 1/11/05 39
SIR for Partial-Response Channels
![Page 40: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/40.jpg)
LANL Workshop 1/11/05 40
Computing the Capacity
• For Markov input process of specified order r , this technique can be used to find the mutual information rate. (Apply it to the combined source-channel.)
• For a fixed order r , [Kavicic, 2001] proposed a Generalized Blahut-Arimoto algorithm to optimize the parameters of the Markov input source.
• The stationary points of the algorithm have been shown to correspond to critical points of the information rate curve [Vontobel,2002] .
![Page 41: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/41.jpg)
LANL Workshop 1/11/05 41
Capacity Bounds for Dicode h(D)=1-D
![Page 42: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/42.jpg)
LANL Workshop 1/11/05 42
Markovian Sufficiency
Remark: It can be shown that optimized Markovian processes whose states are determined by their previous r symbols can asymptotically achieve the capacity of finite-state intersymbol interference channels with AWGN as the order r of the input process approaches ∞.
(This generalizes to 2 dimensional channels.) [Chen-Siegel, 2004]
![Page 43: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/43.jpg)
LANL Workshop 1/11/05 43
Capacity and SIR in Two Dimensions
• In two dimensions, we could estimate by calculating the sample entropy rate of a very large simulated output array.
• However, there is no counterpart of the BCJR algorithm in two dimensions to simplify the calculation.
• Instead, conditional entropies can be used to derive upper and lower bounds on .
( )YH
( )YH
![Page 44: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/44.jpg)
LANL Workshop 1/11/05 44
Examples of Past{Y[i,j]}
![Page 45: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/45.jpg)
LANL Workshop 1/11/05 45
Conditional Entropies
• For a stationary two-dimensional random field Y on the integer lattice, the entropy rate satisfies:
(The proof uses the entropy chain rule. See [5-6]) • This extends to random fields on the hexagonal lattice,
via the natural mapping to the integer lattice.
( ) [ ] [ ]{ }( )j,i,kj,i YPastYHYH | ∞=
![Page 46: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/46.jpg)
LANL Workshop 1/11/05 46
Upper Bound on H(Y)
• For a stationary two-dimensional random field Y,
where
( ) 1Ul,k
kHminYH ≤
( ) [ ] [ ]{ }( )j,il,kj,i YPastYHYH |Ul,k =1
![Page 47: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/47.jpg)
LANL Workshop 1/11/05 47
Two-Dimensional Boundary of Past{Y[i,j]}
• Define to be the boundary
of . • The exact expression for is messy, but the geometrical concept is simple.
[ ]{ }j,il,k YPast
[ ]{ }j,il,kStrip Y
[ ]{ }j,il,kStrip Y
![Page 48: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/48.jpg)
LANL Workshop 1/11/05 48
Two-Dimensional Boundary of Past{Y[i,j]}
![Page 49: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/49.jpg)
LANL Workshop 1/11/05 49
Lower Bound on H(Y)
• For a stationary two-dimensional hidden Markov field Y,
where
and is the “state information” for the strip .
( ) 1Ll,k
kHmaxYH ≥
( ) [ ] [ ]{ } [ ]{ }( )( )j,iYl,kStX,j,il,kj,i YPastYHYH |Ll,k =1
[ ]{ }( )j,iYl,kStX
[ ]{ }j,iYl,kStrip
![Page 50: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/50.jpg)
LANL Workshop 1/11/05 50
Computing the SIR Bounds
• Estimate the two-dimensional conditional entropies over a small array.
• Calculate to get for many realizations of output array.
• For column-by-column ordering, treat each row as a variable and calculate the joint probability row-by-row using the BCJR forward recursion.
( )BAP
( )BAH
( ) ( )BPBAP ,,
iY
{ }mYYYP ,,, 21
![Page 51: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/51.jpg)
LANL Workshop 1/11/05 51
2x2 Impulse Response
• “Worst-case” scenario - large ISI:
• Conditional entropies computed from 100,000 realizations.
• Upper bound:
• Lower bound:
(corresponds to element in middle of last column)
=
5.05.05.05.0
],[1 jih
[ ] [ ] ( )
− 1 ,log
21min 0
10,3,7,7,1,2 eNH U π
[ ] [ ] ( )01
0,3,7,7,1,2 log21 eNH L π−
![Page 52: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/52.jpg)
LANL Workshop 1/11/05 52
SIR Bounds for 2x2 Channel
![Page 53: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/53.jpg)
LANL Workshop 1/11/05 53
Computing the SIR Bounds
• The number of states for each variable increases exponentially with the number of columns in the
array. • This requires that the two-dimensional impulse response
have a small support region.
• It is desirable to find other approaches to computing bounds that reduce the complexity, perhaps at the cost of weakening the resulting bounds.
![Page 54: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/54.jpg)
LANL Workshop 1/11/05 54
Alternative Upper Bound
• Modified BCJR approach limited to small impulse response support region.
• Introduce “auxiliary ISI channel” and bound
where and is an arbitrary conditional probability distribution.
( ) 2Ul,kHYH ≤
[ ] [ ]{ } [ ] [ ]{ } ydjilkjiyqjilkjiypH yPastyPastUlk
−= ∫∫
∞
∞−,,,log,,,, |2
,
[ ] [ ]{ }
jilkjiyq yPast ,,, |
![Page 55: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/55.jpg)
LANL Workshop 1/11/05 55
3x3 Impulse Response
• Two-DOS transfer function
• Auxiliary one-dimensional ISI channel with memory length 4. • Useful upper bound up to Eb/N0 = 3 dB.
=011121110
101],[2 jih
![Page 56: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/56.jpg)
LANL Workshop 1/11/05 56
SIR Upper Bound for 3x3 Channel
![Page 57: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/57.jpg)
LANL Workshop 1/11/05 57
Concluding Remarks
• Recent progress has been made in computing information rates and capacity of 1-dim. noisy finite-state ISI channels.
• As in the noiseless case, the extension of these results to 2-dim. channels is not evident.
• Upper and lower bounds on the SIR of two-dimensional finite-state ISI channels have been developed.
• Monte Carlo methods were used to compute the bounds for channels with small impulse response support region.
• Bounds can be extended to multi-dimensional ISI channels.
• Further work is required to develop computable, tighter bounds for general multi-dimensional ISI channels.
![Page 58: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/58.jpg)
LANL Workshop 1/11/05 58
References
1. D. Arnold and H.-A. Loeliger, “On the information rate of binary-input channels with memory,” IEEE International Conference on Communications, Helsinki, Finland, June 2001, vol. 9, pp.2692-2695.
2. H.D. Pfister, J.B. Soriaga, and P.H. Siegel, “On the achievable information rate of finite state ISI channels,” Proc. Globecom 2001, San Antonio, TX, November2001, vol. 5, pp. 2992-2996.
3. V. Sharma and S.K. Singh, “Entropy and channel capacity in the regenerative setup with applications to Markov channels,” Proc. IEEE International Symposium on Information Theory, Washington, DC, June 2001, p. 283.
4. A. Kavcic, “On the capacity of Markov sources over noisy channels,” Proc. Globecom 2001, San Antonio, TX, November2001, vol. 5, pp. 2997-3001.
5. D. Arnold, H.-A. Loeliger, and P.O. Vontobel, “Computation of information rates from finite-state source/channel models,” Proc.40th Annual Allerton Conf. Commun., Control, and Computing, Monticello, IL, October 2002, pp. 457-466.
![Page 59: Capacity of Noiseless and Noisy Two-Dimensional Channelscmrr-star.ucsd.edu/static/presentations/lanl.pdf · Capacity of Noiseless and Noisy Two-Dimensional Channels . Paul H. Siegel](https://reader033.vdocuments.us/reader033/viewer/2022060221/5f075e457e708231d41ca31d/html5/thumbnails/59.jpg)
LANL Workshop 1/11/05 59
References
6. Y. Katznelson and B. Weiss, “Commuting measure-preserving transformations,” Israel J. Math., vol. 12, pp. 161-173, 1972.
7. D. Anastassiou and D.J. Sakrison, “Some results regarding the entropy rates of random fields,” IEEE Trans. Inform. Theory, vol. 28, vol. 2, pp. 340-343, March 1982.