andrea montanari and ruediger urbanke tifr tuesday, january 6th, 2008 phase transitions in coding,...
TRANSCRIPT
Andrea Montanari and Ruediger UrbankeTIFR
Tuesday, January 6th, 2008
Phase Transitions in Coding, Communications, and Inference
Outline
1) Thresholds in coding, the large size limit
(definition and density evolution characterization)
2) The inversion of limits (length to infty vs size to infty) 3) Phase transitions in measurements (compressed sensing versus message passing, dense versus sparse matrices)
4) Phase transitions in collaborative filtering (the low-rank matrix model)
Model
Shannon ’48
binary symmetric channel
capacity: R≤1-h(ε)
binary erasures channel
capacity: R≤1-ε
Channel Coding
code
decoding
C={000, 010, 101, 111}
n ... blocklength
xMAP(y)=argmaxX in C p(x | y)
xiMAP(y)=argmaxXi p(xi |y)
Factor Graph Representation of Linear Codes
(7, 4) Hamming code
every linear code
Tanner, Wiberg, Koetter, Loeliger, Frey
parity-check matrix
Low-Density Parity Check Codes
(3, 4)-regular codes
Gallager ‘60
number of edges is linear in n
Ensemble
Variations on the Theme
irregular LDPC ensembleregular RA ensembleirregular MN ensembleirregular RA ensembleARA ensembleturbo code
degree distributions as well as structure
protographirregular LDGM ensemble
(Luby, Mitzenmacher, Shokrollahi, Spielman, and Stehman)Divsalar, Jin, and McEliece Jin, Khandekar, and McEliece Abbasfar, Divsalar, KungBerrou and GlavieuxThorpe, Andrews, DolinarDavey, MacKay
Message-Passing Decoding -- BEC
?
?
00
0
?
?
?
0+?0+? =??
0
0
?
?
?
??
0=00?
?
0
0
0
?0
?
decoded
decoded
0+00+0 =00
Message-Passing Decoding -- BSCGallager Algorithm
Asymptotic Analysis: Computation Graph
probability that computation graphof fixed depth becomes tree
tends to 1 as n tends to infinity
Asymptotic Analysis: Density Evolution -- BEC
x
1-(1-x)r-1
x x
ε (1-(1-x)r-1)l-1
ε
Luby,Mitzenmacher, Shokrollahi,
Spielman, and Steman ‘97
Asymptotic Analysis: Density Evolution -- BEC
ε
phase transition: εBP so that xt → 0 for ε< εBP
xt → x∞>0 for ε> εBP
Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm
phase transition: εBP so that xt → 0 for ε< εBP
xt → x∞>0 for ε> εBP
xt =ε (1-p+(xt-1))+(1-ε) p-(xt-1)
p+(x)=((1+(1-2x)r-1)/2) l-1 p-(x)=((1-(1-2x)r-1)/2) l-1
Asymptotic Analysis: Density Evolution -- BP
Inversion of Limits
size versus number of iterations
Density Evolution Limit
Density Evolution Limit
“Practical” Limit
“Practical” Limit
The Two Limits
Easy: (Density Evolution Limit)
Hard(er): (“Practical Limit”)
Binary Erasure Channel
DE Limit
“Practical” Limit
implies
What about “General” Case
expansion
probabilistic methods
Korada and U.
Expansion
Miller and Burshtein: Random element of LDPC(l, r, n) ensemble is expander with
expansion close to 1-1/l with high probability
expansion ~ 1-1/l
Why is Expansion Useful?
Setting: Channel
Setting: Ensemble
Setting: Algorithm
Aim: Show for this setting that ...
DE Limit
“Practical” Limit
implies
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
Linearized Decoding Algorithm
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
Combine with Density Evolution
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
Correlation and Interaction
0 1
1 000Expected growth:
(r-1) 2 ε?< 1
Problem: interaction correlation
(r-1)
2 ε
Correlation and Interaction
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
Witness
Witness
Witness
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
Monotonicity
Randomizing the Noise Outside
randomizing noise outside the witness increases the probability of error
FKG
→
←⁄
≤
Proof Outline
linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the
witness sub-critical birth and death
process
Expansion
random graph has expansion close to expansion of a treewith high probability
⇒this limits interaction
0 1
1 000
References
For a list of references see:http://ipg.epfl.ch/doku.php?id=en:courses:2007-2208:mct
Results
Open Problems
0.0
0.4
0.3
0.2
0.1
0.2 0.4 0.6 0.8
Pb
channel entropy