error correction on a tree: instanton approach misha chertkov (lanl) in collaboration with: v....
TRANSCRIPT
Error correction on a tree: Instanton approach
Misha Chertkov (LANL)
In collaboration with: V. Chernyak (Corning) M. Stepanov (UA, Tucson) B. Vasic (UA, Tucson)
Thanks:I. Gabitov (Tucson/LANL)
Boulder: 04/15/04
Submitted to Phys.Rev.Lett.
•Forward-Error-Correction (FEC). Channel Noise. •Coding. Low Density Parity Check codes (LDPC) - Tanner graph •Decoding. Marginal-A-Posteriori (MAP) – Stat Mech interpretation Belief Propagation (BP) – Message Passing (MP) •Post-Error-Correction Bit-Error-Rate (BER). Optimization. Shannon transition/limit . Error floor - Evaluation.
•Tree as an approximation: BP is exact . From LDPCC to a tree •BER in the center of the tree • High Signal-to-Noise Ratio (SNR) phase. Hamming distance. •Symmetry. * Broken Symmetry. *•Instantons/phases on the tree.
Introduction:
What is next? *
Our objectives
L
L
N
N
Forward-Error-Correction
Coding
Decoding
N > L R=L/N - code rate
)|()|( )(
1
)()()( ini
N
i
outi
inout xxpxxP
22exp)|( 2 syx
syxp
channelwhite
Gaussian symmetricexam
ple
menu
Nxxx ,,1
)()()( inoutin xxxnoise
Low Density Parity Check Codes
menu
N=10variable nodes
M=N-L=5 checking nodes
Parity check matrix
0
0
0
0
0
10
9
8
7
6
5
4
3
2
1
x
x
x
x
x
x
x
x
x
x
H mod 2
Tanner graph
M
ii
ii x
1
1,
112
“spin” variables -
- set of constraintsM
Ni
,,1
,,1
(linear coding)
Decoding (optimal)
N
kkk
M
ii hhFhZ
11}{
exp1,)(exp)(
1sh“magnetic” field(external/noise)
)|( )()( inout xxP
constraints
“free energy”
“statistical sum”
menu
(symbol to symbol) Maximum-A-Posteriori (MAP) decoding
)()( hmsignhoutput j
Efficient but Expensive:requires operationsL2
hhFhm
)()(
“magnetization”
Stat Mech interpretation was suggested byN. Sourlas (Nature ‘89)
To notice – spin glass (replica) approach for random codes:e.g. Rujan ’93, Kanter, Saad ’99; Montanari, Sourlas ’00; Montanari ’01; Franz, Leone, Montanari, Ricci-Tersenghi ‘02
Sub-optimal but efficient decoding
i
jii
j
jj
i
jii
j
jj
hm
h
tanhtanhtanh
tanhtanh
1
1 Belief Propagation (BP) Gallager’63;Pearl ’88;MacKay ‘99
=solving Eqs. on the graph
it
i
i
ji
ti
j
jt
j
h
h
)(
)(1)1( tanhtanh
Iterative solution of BP= Message Passing (MP)
Q*m*N steps instead of Q - number of MP iterations
m - number of checking nodes contributing a variable node
L2
What about efficiency? Why BP is a good replacement for MAP?
* (no loops!)
menu
Post-Error-Correction Bit Error Rate (BER)
2/1
20
1
)2(2
)(exp)( N
N
j j
ii ss
shhmdhdB
measure for unsuccessful decoding
Probability of making an error in the bit “i”
{+1} is chosen for the initial code-word
probability density for givenmagnetic field/noise realization
Foreword-error-correction scheme/optimizationForeword-error-correction scheme/optimization
1. describe the channel/noise --- External2. suggest coding scheme3. suggest decoding scheme4. measure BER/FER5. If BER/FER is not satisfactory (small enough) goto 2
menu
From R. Urbanke, “Iterative coding systems”
menu
SNR, s
BE
R, B
Shannon transition/limit
Error floor
Error floor prediction for some regular (3,6) LDPC Codes using a 5-bit decoder. From T. Richardson “Error floor for LDPC codes”, 2003 Allerton conference Proccedings.
menu
No-go zone for brute-force Monte-Carlo numerics.
Estimating very low BER is the major bottleneck of the coding theory
Our objective:
For given (a) channel (b) coder (c) decoderto estimate BER/FER by means ofanalytical and/or semi-analytical methods.
Hint:
BER is small and it is mainly formed at some very special“bad” configurations of the noise/”magnetic field”
Instanton/saddle-point approach is the right way to identifythe “bad” configurations and thus to estimate BER!
menu
menu
Tree -- no loops -- approximation
}{}{
1
}{
}{}{
1
}{
11}{
exp1,)(
exp1,)(
exp1,)(exp)(
kkk
iij
kkk
iij
N
kkk
M
ii
hhY
hhX
hhFhZ
j
j
i
jii
j
jj h tanhtanh 1
2/)/ln(
)()(2
1)exp(
)()(2
1)exp(
jjj
i
ji
i
jiiiii
j
jj
i
ji
i
jiiiii
j
jj
XY
YXYXhY
YXYXhX
MAP
BP
Belief Propagation is optimal (i.e. equivalentto Maximum-A-Posteriori decoding) on a tree (no loops)
Analogy: Bethe lattice (1937)
Gallager ’63; Pearl ’88; MacKay ’99Vicente, Saad, Kabashima ’00; Yedidia, Freeman, Weiss ‘01
From a finite-size LDPCC to a tree:
1) Fix the variable node where BER needs to be calculated2) Choose shortest loop on the graph coming through the “0”th
node. Length of the loop is (n+1). 3) Count n-generations from the tree center and cut the rest.
Regular graph/tree is characterized by:m - number of checking nodes connected to a variable nodek - number of variable nodes connected to a checking noden - number of generations on the tree
m=2,k=3,n=4menu
BER in the center of the tree
0
tanh2
1tanh
2
1
)(exp
2
1
2
0 0
1
0
1 00
Q
ss
ss
Q
QddB
j
j
k
jkkj
k
kk
jj
Tree is directed thus integrating over the ``magnetic fields” one gets a path-integral over new fields, , defined on the variable nodes.
j
menu
Remarks: 1) Optimal configuration/instanton depends on SNR, s; 2) There are may be many competing instantons; 3) Looking for instantons pay attention to the symmetry
2/1
20
1
)2(2
)(exp)( N
N
j j
ii ss
shhmdhdB
i
jii
j
jj
i
jii
j
jj
hm
h
tanhtanhtanh
tanhtanh
1
1
Instanton equations!
Effective action
High Signal-to-Noise-Ratio (SNR) phase
menu
Original code word = “+1” on the entire tree
The next “closest” code word = “-1” on the colored branches, = “+1” on the remaining variable nodes
Hamming distance between the two code words= number of the colored variable nodes
2exp0
HsB
at s>>1 That is also given by an instanton:
0 node is colored
node is not colored
Analogy with a low-temperature phase in stat-mech:High SNR value of effective action ~ self energy
Low SNR -- symmetric -- phase
menu
Symmetric phase: at any node on the tree depends
primarily on the generation (counted from the center)
11
2112
1
1
0
10
tanhtanh)1(),(
2
),(),())1)(1((
2
)1(
lm
nmjmj
n
j
jn
mssg
sgsglm
lmQ
);,('),(),()1(
),('),(),()1)(1(
1121
11
sgsgsglm
sgsgsglm
nmnmnmn
jmjmjjmj
01
for j=0,…,n-2instantonequations
“zero momentum” configuration/approximation
s
sg jmj
0
1 ),(
guarantees estimation
from above for eff. action
),('1
),(
ccm
ccmc
sg
sg
Shannon’s
transition
0
21
;0 2
)),((Q
sgQ nm
zm
c
c
ss
ss
finite
infinite
“0” “2”“1”
“4”“3”
In general:There are many (!!!) broken symmetryinstanton solutions
Remark:Broken symmetry instantons may be related
to the “near codewords” suggested by Richardson ‘03 in the context of the error-floor phenomenon explanation menu
Broken symmetry
High SNR(low temp)
Low SNR(high temp)
m=4, l=5, n=3. Curves of different colors correspond to
the instantons/phases of different symmetries.
Instanton phases on the tree
menutruth …
sg
msg
sgssk
sss
ms
lm
ccmcck
n
,1
tanhtanh)1(),(
,:11
121
transitions
Full numerical optimization (no symmetry breaking was assumed !!!)
Area of a circle surrounding any variable node is proportional
to the value of the noise on the node.
m=2 l=3 n=3
menu
What is next?
We plan to develop and extend this instanton approach to:
•Regular codes with loops. This task will require developing a perturbation theory with respect to the inverse length of the closed loop and/or with respect to the small density of closed loops. •Other types of codes, e.g. convolutional, turbo, etc.•Calculation of the Frame Error Rate (FER), thereby measuring the probability of making an error in a code word.•Finite-number of iteration in message-passing version of the BP algorithm. The particular interest here lies in testing how BER in general and the error floor phenomena in particular depend on the number of iterations. •Other types of fast but, probably, less efficient decoding schemes.•Other types of uncorrelated channels (noise), e.g. binary eraser channel.•Correlated channels, with both positive and negative types of correlations between neighboring slots. This is particularly relevant for linear and nonlinear (soliton) transmission in fiber optics communications.•Accounting for Gaussian fluctuations (i.e. second order effects) around the instantons.
menu
Truth …
menu
mainslide