constructing error-correction codes from scale-free networks

Constructing ErrorConstructing Error--Correction Codes Correction Codes from Scalefrom Scale--Free NetworksFree Networks

Francis C.M. LauFrancis C.M. Lau

Department of Electronic and Information EngineeringHong Kong Polytechnic University

International Workshop on Complex Systems and Networks 2007 Guilin, China

TTHEHE HHONGONG KKONGONG

PPOLYTECHNICOLYTECHNIC UUNIVERSITYNIVERSITY

IWCSN' 2007, Guilin, China 2



Part 1: Communications and CodingPart 1: Communications and Coding




Communications without CodingHow are you today ?

Hxw au& u%$ wqo .

Welf affi zv iol bxg.

How aruyox tuday ?

Information can be easily corrupted when sent through a channel !




More Reliable Communications

How are you today ?How are you today ?How are you today ?

How aru yox tuday ?How aee yeu todey ?Hoe are you toxak ?

How are you today ?

Error-correction capability

channel




Bit-level Communications

111100

011010

101001

110000

10

Without coding schemes:

0,1 0,0

InfoSource

channel InfoSink

noise

0,1 0,1000, 111 000, 110

1 information bit2 check bits

Code Rate=number of information bits/ block length=1/3

Block Length =1+2=3

With coding schemes:Info

SourceEncoder channel Decoder Info

Sink

noise




Reliable Communications

Add redundant information at transmitterDecode information intelligently at receiver

Error-Correction Capability

Any better ways than to repeat the information several times?Any performance bounds?




Shannon’s Capacity Theorem

Channel

Additive White Gaussian NoiseBandwidth W

Average Received Signal Power SAverage Noise Power N

⎟⎠⎞

⎜⎝⎛ +=

NSWC 1log2

System Capacity of the channel

C. E. Shannon “A mathematical theory of communications,” Bell Syst. Tech. J., vol. 27, pp. 379–423, 623–656, 1948.




Shannon’s Capacity Theorem

possible theoretically to transmit information at any rate R ≤ C with an arbitrarily small error probability (with coding)if R > C, not possible to transmit information with an arbitrarily small error probability (even with coding)

⎟⎠⎞

⎜⎝⎛ +=

NSWC 1log2

ChannelInformation with rate R




Normalized channel capacity versus SNR

⎟⎠⎞

⎜⎝⎛ +=

NS

WC 1log2




Normalized channel bandwidth versus SNR

⎟⎠⎞

⎜⎝⎛ +

=

NSC

W

1log

1

2

The graph is not telling the whole story !




Because …

noise power is proportional to bandwidth

WNN 0=

⎟⎠⎞

⎜⎝⎛ +

=

NSC

W

1log

1

2




Further …

when bit rate R equals channel capacity C

CS

RSEb ==energy per bit




Shannon Limit

⎟⎠⎞

⎜⎝⎛ +=

NS

WC 1log2

( )12 /

0

−= WCb

CW

NE

WNN 0=

CS

RSEb ==

rearrangement




( )12 /

0

−= WCb

CW

NE

Shannon Limit

0//

dB 59.1/ 0

→⇔∞→⇒−→

WCCW

NEb

Channel capacity approaches zero, regardless of the channel bandwidth

No error-free communications below dB 59.1/ 0 −=NEb




Typical error performance of coded and uncodedmodulations




Improved error performance;more bandwidth required to

add redundancy bits




Reduction in requirement;more bandwidth required to add

redundancy bits

0/ NEb

coding gain




Significance/Conclusions of Shannon’s work

proved theoretically that there exists codes that could improve the error probability performance from uncoded modulation schemesthere is a minimum requirement0/ NEb




BUT ….




How should we design coding schemes, with reasonable

complexity, that work as close to the Shannon limit as possible?




Solutions

Not provided by Shannon !

So do research on Coding !




Part 2: ParityPart 2: Parity--Check CodesCheck Codes




Parity-Check Codes

single-parity-check code

110011010100

parity bit

message bits




Parity-Check Codes

single-parity-check code

even-parity code

can detect all single-and triple-error patterns (e.g. 0100 or 0010) but cannot correct errors




001111110011110000000110100001101011

Parity-Check Codes

rectangular code (or product code)

horizontal parity check

vertical parity check

message




001111110011110000000110100001101011

Parity-Check Codes

rectangular code (or product code)

can correct a single error pattern

001111110011110000001110100001101011

horizontal parity check fails

vertical parity check fails

bit in error

channel




Linear Block Codes

a class of parity-check codesdenoted by (n, k)

codeword length message length

maps k-bit messages (k-tuples) linearlyand uniquely to n-bit codewords (n-tuples)




Subset S of a vector space is a subspace if

it contains the all-zeros vectorsum of any two vectors in S is also in S




Packing as many Packing as many codewordscodewords in the entire in the entire space as possible improves space as possible improves coding efficiency coding efficiency

Putting the Putting the codewordscodewords as as far apart from one far apart from one another as possible another as possible increases the chance of increases the chance of decoding the decoding the codewordscodewordscorrectlycorrectly




(6, 3) Code Example

form a subspace

011101+

Modulo-2 Addition

all-zeros vector




Modulo-2 Multiplication• 0 10 0 01 0 1

Modulo-2 Addition and Multiplication

addition can be accomplished electronically using an Exclusive-OR gatemultiplication can be accomplished using an AND gate

Modulo-2 Addition+ 0 10 0 11 1 0




Encoding the Messages

Table look-up is possible for small kFor large k, table look-up may become extremely difficult

e.g., 301026.12100 ×≈⇒= kk

Use of Generator Matrix




Generator Matrix G (size k x n)

a basis set of k linearly independent n-tuples that spans the subspace

n-tuple

n-tuple

n-tuple

][ 21 kmmm L=mmessage

mGU =codeword (size 1 x n)




(6, 3) Code Example

][ 654321 uuuuuu=




Parity-Check Matrix H (size (n-k) x n)

For each generator matrix G, there exists an (n-k) x nmatrix H such that rows of G are orthogonal to rows of H.

0GH =Tk x (n-k) all-zeros matrix

0mGHUH == TT

H can be used to test whether a received vector is a valid codeword.

HG ↔





0GH =T

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

110100011010101001

H

rows areorthogonal




Parity-Check Matrix H

000

101110011100010001

][

653

542

641

654321

=++=++=++

⇒

=

⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

⇒=

uuuuuuuuu

uuuuuuT 00UH




Bipartite Graph

)0(5422 =++= uuuc

)0(6533 =++= uuuc

1u

2u

3u

4u

5u

6u

variable nodes

check nodes

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

110100011010101001

H

)0(6411 =++= uuuc




Decoding

received vector r’

Codeword UAdditive White Gaussian Noise Channel0 +1 volt

1 −1 volt




Eight codewords in a 6-tuple space

Hard Decoding

received vector r’(AWGN channel)

decoded codeword after error correction

after making hard decision on each bit

ri > 0 volt 0ri < 0 volt 1




r’

)'|011101( rU =P

)'|101110( rU =P

)'|110100( rU =P

maximuma posteriori (MAP) decision rule: Select codeword U that has the largest )'|( rUP

)'|101001( rU =Pa posteriori probability (APP)

Soft Decoding




Performance of Some Well-known Block Codes (Coherent BPSK over an AWGN channel)

t = maximum number of guaranteed correctable errors per codeword




Performance of BCH Codes (Coherent BPSK over an AWGN channel)




Part 3: LowPart 3: Low--DensityDensity--ParityParity--Check (LDPC) CodesCheck (LDPC) Codes





0GH =T

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

110100011010101001

H




Low-Density-Parity-Check Codes

proposed by Gallager(1960)parity-check matrix H

sparse (most elements are zeros)fraction of 1’s ~ O(n)

elements of H determine the connections between variable nodes and check nodes degree of variable

node u6 = 2

degree of checknode c3 = 3




Low-Density-Parity-Check Codes

sparse (low-density) parity-check matrix Himplies that all variable nodes and check nodes have very few connections

HG ↔




Error rates achieved by different coding schemes under the binary AWGN channel.Codeword length = 106. Rate =0.5.




Types of LDPC Codes

Regular LDPC all nodes of the same type (variable node or check node) have the same degree

A (3, 6)-regular LDPC code of length 10 and rate one-half.

check node degree

variable node degree




Types of LDPC Codes

Irregular LDPC: the degrees of each set of nodes are chosen according to some distribution




Degree Distribution of Nodes

Degree distribution of variable nodes

Degree distribution of check nodes

fraction of edges connected to the variable nodes with degree k

∑=

−=vd

k

kk xx

2

1)( λλ

∑=

−=cd

k

kk xx

2

1)( ρρfraction of edges connected to the

check nodes with degree k

maximum variable node degree

maximum check node degree




Code Rate

∫∫−= 1

0

1

0

d)(

d)(1

xx

xxR

λ

ρ




Question

Given specific

∑=

−=vd

k

kk xx

2

1)( λλ ∑=

−=cd

k

kk xx

2

1)( ρρand .

How would the LDPC code perform?




AnswerDependent on the actual design

(connections),

the channel type, e.g. AWGN, binary symmetric channel (BSC),

binary erasure channel (BEC)

and the decoding algorithm.




More Question

Any idea on the optimal performance of

practical LDPC decoders?




Part 4: Part 4: Belief Propagation (BP) Decoding Belief Propagation (BP) Decoding AlgorithmAlgorithm




Belief Propagation (BP) Decoding Algorithm

A kind of message-passing decoding algorithmApplicable to both regular and irregular LDPC codesProduces very good error performance




Belief Propagation (BP) Decoding Algorithm

Define Log Likelihood Ratio (LLR):

⎥⎦

⎤⎢⎣

⎡==

info)|1bit(info)|0bit(log

PP

variable nodes

check nodes




BP Decoding Algorithm

Compute initial Log Likelihood Ratio (LLR) for each variable node based on the received signal vector r (real number elements)

⎥⎦

⎤⎢⎣

⎡==

)|1bit()|0bit(log

i

i

rPrP

1r)(LLR 10 r

iterationnumber

ir





Set iteration number k = 1Pass the LLR messages from variable nodes to the connected check nodes

)(LLR 10 r





Check nodes received the LLR messages from the connected variable nodes





Each check node computes a message for each of its connected variable nodes, based on the messages from all other variables nodes





Each variable node update its LLR based on the messages passed from the check nodes and the initial LLRBased on the updated LLR, estimate the codeword

)(LLR)(LLR 1110 rr →





Estimate the codeword as

)(LLR)(LLR 1110 rr →

[ ]nccc ˆˆˆˆ 21 L=c

where ⎭⎬⎫

⎩⎨⎧

<>

=0)(LLR if1 0)(LLR if0

ˆ1

1

rr

ck

ki

If , is the decoded codeword.

0Hc =Tˆ c





)(LLR 10 r

If , increment the iteration number k.Each variable node computes a message for each of its connected check nodes, based on its initial LLR and the messages from all other connected check nodes

0Hc ≠Tˆ





)(LLR 10 r

Each variable node computes a message for each of its connected check nodes, based on its initial LLR and the messages from all other connected check nodes





Check nodes received the LLR messages from the connected variable nodes





Each check node computes a message for each of its connected variable nodes, based on the messages from all other variables nodesSame iterative process repeated …. until convergence to a valid codeword or maximum number of iterations exceeded




Capacity (Threshold) of LDPC codesGiven the degree distributions

and the channel type (AWGN, BSC or BEC)and the use of BP decoding algorithm.

∑=

−=vd

k

kk xx

2

1)( λλ ∑=

−=cd

k

kk xx

2

1)( ρρ

Richard and Urbanke (2001) proposed an effective algorithm – density evolution – to determine the capacity (threshold) of LDPC codes.

T. J. Richardson and R. Urbanke, “The capacity of low-density parity-check codes under message-passing decoding,” IEEE Trans. Inform. Theory, vol. 47, pp. 599–618, Feb. 2001.




Density Evolution Algorithm

channel type

∑=

−=vd

k

kk xx

2

1)( λλ

∑=

−=cd

k

kk xx

2

1)( ρρDensity

Evolution Algorithm

iterations

threshold value *σ

a higher threshold value indicates a higher achievableachievable performance of the code




Good Degree Distribution Pairs (Rate = 0.5)

T. J. Richardson et al., “Design of capacity-approaching irregular low-density parity-check codes,”IEEE Trans. Inform. Theory, vol. 47, pp. 619–637, Feb.2001.




Good Degree Distribution Pairs (Rate = 0.5)




the threshold value , for example indicates the maximum noise power that can be tolerated for error-free communication in AWGN channelsthe threshold value can be achievedachieved if

the message-passing process does not contain any cyclesnumber of iterations tends to infinitycodeword length is infinitecodeword length is infinite

Density Evolution Algorithm*σ

*σ




Density Evolution AlgorithmProblems:

optimizing the codes based on DE algorithm is not a simple task

codeword length cannot be infiniteoptimizing the threshold value may give a more complex code

number of connections

∑=

−=vd

k

kk xx

2

1)( λλ ∑=

−=cd

k

kk xx

2

1)( ρρ

vary to maximize the threshold value




Part 5: Review of Complex NetworksPart 5: Review of Complex Networks




Basic properties

Path length: the distance between two nodes, which is defined as the number of edges along the shortest path connecting them




Basic properties

Betweenness centrality: the fraction of shortest paths going through a given nodeAssortative mixing: preference of high-degree nodes attach to other high-degree nodesDisassortative mixing: preference of high-degree nodes attach to low-degree nodes




Examples of Complex Networks

Random NetworksGiven a network with N nodes. Each pair of nodes are connected with a probability of p. Poisson distribution

( )!

kep kk

μμ −

=

μ =

0.1ERp =





Regular Coupled Networks

high clusteringlarge average path length

Fully-connected Networks

2 2.5 3 3.5 4 4.50

0.2

0.4

0.6

0.8

1

<k>

P(k

)





Small-World NetworksEach edge of a regular coupled network is re-wired with a probability of phigh clusteringsmall average path length

0WSp =

0.2WSp =





Scale-Free Networks




Examples of Complex NetworksScale-Free (SF) Networks

γii nn −~)Pr( γxxf −~)(




Characteristics of Typical Complex Networks

HighUniformLongRegular Coupled Network

-

High

Low

Clustering Coefficient

Power-LawVery ShortScale Free Networks

-Short Small World Networks

PoissonShortRandom Networks

Degree Distribution

Average Distance

(log( ))NΟ

(log(log( )))*NΟ

*The exponent parameter should be valued between 2 and 3.See reference “Scale-Free Networks Are Ultrasmall”, PRL, vol. 90, no. 5

(log( ))NΟ

( )NΟ

Fast to disseminate information




Part 6: ScalePart 6: Scale--free Networks to free Networks to LDPC CodesLDPC Codes




Scale-free Networks meet LDPC Codes

Can the “very short distance” property of scale-free network helps passing/spreading messages quickly in the decoding of LDPC codes?

If so, how?




Scale-free Networks meet LDPC Codes

??




From Bipartite Graph to Unipartite Graph




From Bipartite Graph to Unipartite Graphpower-law degree distribution

Power-law degree distribution !




Typical node degree distribution of the corresponding unipartite graph. Codeword length = 10000 and maximum variable node degree = 20.X. Zheng, F.C.M. Lau and C.K. Tse, " Study of LDPC Codes Built on Scale-Free Networks," Proceedings, NOLTA'06, Bologna, Italy, September 2006, pp. 563-566.




Building LDPC Codes From SF Networks

Assume that the variable nodes have the power-law degree distribution and the check nodes obey the Poisson-law degree distribution

Use the (Density Evolution) DE to select the optimized parameters and .

( ) ~P k k γλ

−

γ μ

!)(

lelP

l μ

ρμ −

=




Threshold value and average variable node degrees <k> of LDPC codes built from scale-free networks and the optimized ones reported in [1] for an AWGNchannel. Rate equals 0.5.

*σ

[1]

[1] T. J. Richardson et al., “Design of capacity-approaching irregular low-density parity-check codes,”IEEE Trans. Inform. Theory, vol. 47, pp. 619–637, Feb.2001.




Typical node degree distribution of the corresponding unipartite graph. Codeword length = 1000 and maximum variable node degree = 15.

X. Zheng, F.C.M. Lau and C.K. Tse, “Error Performance of Short-Block-Length LDPC Code Built on Scale-Free Networks,” Proceedings, The Third Shanghai International Symposium on Nonlinear Sciences and Applications, Shanghai, China, June 2007, pp. 55-57.




Building LDPC Codes From SF Networks

Threshold values lower compared with those reported in the literatureAverage variable node degrees <k> lower compared with those reported in the literature

*σ

Which one is better in practice ?




The threshold value can be achievedachieved ifthe message-passing process does not contain any cycles andnumber of iterations tends to infinity andcodeword length is infinitecodeword length is infinite

Threshold Value

*σ




PEG and Enhanced PEG

Progressive Edge-Growth algorithm (PEG)an effective method to construct codes with girth average as large as possible based on the given degree distributions

Enhanced PEG (E-PEG) proposed by usstopping set and the near codeword are also checked after each variable node is added.

X. Y. Hu, E. Eleftheriou and D. M. Arnold, “Regular and irregular progressive edge-growth tanner graphs,” IEEE Trans. Inform. Theory, vol. 51, no. 1, pp. 386–398, 2005.




Part 7: Simulation ResultsPart 7: Simulation Results




Block Error RatesBlock length=1008Code rate=0.5Max. no. of iterations = 50




Bit Error Rates

Block length=1008Code rate=0.5Max. no. of iterations = 50




0 0.5 1 1.5 2 2.5 310-6

10-5

10-4

10-3

10-2

10-1

100

SNR(dB)

Blo

ck E

rror R

ate

DE10 (PEG), <k>=3.66 DE10 (E-PEG), <k>=3.66

PEG and E-PEG Algorithms





0 0.5 1 1.5 2 2.5 310-8

10-6

10-4

10-2

100

SNR(dB)

Bit

Erro

r Rat

e

DE10 (PEG), <k>=3.66 DE10 (E-PEG), <k>=3.66






0 0.5 1 1.5 2 2.5 310-7

10-6

10-5

10-4

10-3

10-2

10-1

100

SNR(dB)

Blo

ck E

rror R

ate

SF20 (PEG), <k>=3.72 SF20 (E-PEG), <k>=3.72






0 0.5 1 1.5 2 2.5 310-8

10-6

10-4

10-2

100

SNR(dB)

Bit

Erro

r Rat

e

SF20 (PEG), <k>=3.72 SF20 (E-PEG), <k>=3.72






Part 8: SummaryPart 8: Summary




Summary

Coding for a reliable communicationOperation principles of parity-check codesLow-density-parity-check (LPDC) codes Belief propagation decoding algorithm Density evolution




SummaryLPDC codes with scale-free variable-node-degree distribution achieve very good theoretical threshold (error correction performance)Short LPDC codes built with scale-free variable-node-degree distribution outperform other well-known LPDC codes with similar complexity




Some of Our Related Work1. X. Zheng, F.C.M. Lau and C.K. Tse, “Error Performance of Short-Block-Length

LDPC Code Built on Scale-Free Networks,” Proceedings, The Third Shanghai International Symposium on Nonlinear Sciences and Applications, Shanghai, China, June 2007, pp. 55-57.

2. X. Zheng, F.C.M. Lau, Chi K. Tse and S.C. Wong, “Study of Bifurcation Behavior of LDPC Decoders", International Journal of Bifurcation and Chaos, vol. 16, no. 11, pp. 3435-3449, Nov. 2006.

3. X. Zheng, F.C.M. Lau and Chi K. Tse, “Study of LDPC Codes Built on Scale-Free Networks,” Proceedings, International Symposium on Nonlinear Theory and Its Applications (NOLTA'06), Bologna, Italy, September 2006, pp. 563-566.

4. X. Zheng, F.C.M. Lau, C.K. Tse and S.C. Wong, “Techniques for Improving Block Error Rate of LDPC Decoders,” Proceedings, IEEE International Symposium on Circuits and Systems (ISCAS'06), Kos, Greece, May 2006, pp. 2261-2264.

5. X. Zheng, F.C.M. Lau, Chi K. Tse and S.C. Wong, “Study of Nonlinear Dynamics of LDPC Decoders", Proceedings, European Conference on Circuit Theory and Design (ECCTD ‘2005), Dublin, Ireland, August 2005, paper 207. (CD version)




Collaborators

Dr Wai-man TAMProf. Chi K. TSEDr Siu C. WONGMiss Xia ZHENG

Constructing ErrorConstructing Error--Correction Codes Correction Codes from Scalefrom Scale--Free NetworksFree Networks

Francis C.M. LauFrancis C.M. Lau

Department of Electronic and Information EngineeringHong Kong Polytechnic University

International Workshop on Complex Systems and Networks 2007 Guilin, China



Thank You !

constructing error-correction codes from scale-free networks

Documents