mathematical structures of belief propagation algorithms in probabilistic information processing

24 November, 201124 November, 2011 National Tsin Hua University, TaiwanNational Tsin Hua University, Taiwan 11

Mathematical Structures of Mathematical Structures of Belief Propagation Algorithms in Belief Propagation Algorithms in

Probabilistic Information ProcessingProbabilistic Information Processing

Kazuyuki TanakaKazuyuki Tanaka

Graduate School of Information Sciences, Graduate School of Information Sciences, Tohoku University, Sendai, JapanTohoku University, Sendai, Japan

http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/


ContentsContents

1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Various Applications of Probabilistic Various Applications of Probabilistic

Information ProcessingInformation Processing7.7. SummarySummary

National Tsin Hua University, Taiwan 3

Computational model for information processing in data with uncertainty

Probabilistic Inference

Probabilistic model with graphical structure （ Bayesian network ）

Probabilistic information processing can give us unexpected capacity in a system constructed from many cooperating elements with randomness.

Inference system for data with uncertainty

modeling

Node is random variable.Arrow is conditional

probability.

Mathematical expression of uncertainty=>Probability and Statistics

Graph with cycles

Important aspect24 November, 2011


Computational Model for Probabilistic Information Processing

Probabilistic Information Processing Probabilistic Model

Bayes Formula

Algorithm

Monte Carlo MethodMarkov Chain Monte Carlo

MethodRandomized Algorithm

Approximate MethodBelief PropagationVariational Bayes MethodExpectation Propagation

Randomness and Approximation

24 November, 2011


ContentsContents



24 November, 2011 National Tsin Hua University, Taiwan 6

Joint Probability and Conditional Probability

)()|(),( s.t.

PrPr,Pr

)|(Pr

,PrPr

bPabPbaP

aAaAbBbBaA

abPaA

bBaAaAbB

a

b

Conditional Probability of Event A=a when Event B=b has happened.

Probability of Event A=a )(}Pr{ aPaA Joint Probability of Events A=a and B=b

),()()(Pr,Pr baPbBaAbBaA

Random Variable

State Variable

Probability Distribution

Joint Probability Distribution


Joint Probability and Independency of Events

PrPr bBaAbB a

b

In this case, the conditional probability can be expressed as

Events A and B are independent of each other

bBaAbBaA PrPr,Pr

a

b

1Pr

1Pr0Pr

MAbB

AbBAbB


Marginal Probability

1

0

,PrPrM

a

bBaAbB

Let us suppose that the sample space is expressed by Ω= (A=0) (∪ A=1) … (∪ ∪ A=M1) where every pair of events is exclusive of each other.

Marginal Probability of Event B=b in Joint Probability Pr{A=a,B=b}

Marginalize

aa b

a

bBaAbB ,PrPrSimplified Notation

Summation over all the possible events in which every pair of events are exclusive of each other.

aa ba

=

Message

Graph with Two Nodes and One Edge

24 November, 2011 9

Marginal Probabiilty of High-Dimentional Joint Probabilty

Marginalization with respect to c and d

aa bb

cc dd

Hyperedge

a b

c da c d

Hypergraph

Message

c d

dDcCbBaAbBaA ,,,Pr,Pr

aa bb

cc dd

Hyperedge

aa bb

cc ddc d

Hypergraph

Message

National Tsin Hua University, Taiwan

a c d

dDcCbBaAbB ,,,PrPr

Marginalization with respect to a, c and d


Bayes Formulas

aA

aAbBbBaA

Pr

Pr,Pr

bB

aAaAbB

bB

bBaAbBaA

Pr

PrPr

Pr

,PrPr

a

b

bBbBaAbBaA PrPr,Pr

a

a

aAaAbB

bBaAbB

Pr|Pr

,PrPrPrior Probability

Posterior Probability

Marginal Likelihood

Bayesian Network


ContentsContents




Image Restoration by Probabilistic Model

Original Image

Degraded Image

Transmission

Noise

Likelihood Marginal

PriorLikelihood

Posterior

}ageDegradedImPr{

}Image OriginalPr{}Image Original|Image DegradedPr{

}Image Degraded|Image OriginalPr{

Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability.

Bayes Formula24 November, 2011


Prior Probability in Probabilistic Image Processing

EjiEjiji xxxX

},{},{

2

2

1expPr

xi xj

x1 x2 x3 x4

x5 x6 x7 x8

x9 x10 x11 x12

xi xj

2

2

1exp ji xx

}12,11,10,9,8,7,6,5,4,3,2,1{V

}12,8{},11,7{},10,6{},9,5{},8,4{},7,3{},6,2{},5,1{

}12,11{},11,10{},10,9{},8,7{},7,6{},6,5{},4,3{},3,2{},2,1{E

State Variable of Light Intensity at i-th Pixel in Original Image

xi


Additive White Gaussian Noise

ViVi

ii yxxXyY 222

1expPr

Conditional Probability of Degradation Process

X1 X2 X3 X4

X5 X6 X7 X8

X9 X10 X11 X12

y1 y2 y3 y4

y5 y6 y7 y8

y9 y10 y11 y12

xi

222

1exp ii yx

State Variable of Light Intensity at i-th Pixel in Original Imagexi

yi

State Variable of Light Intensity at i-th Pixel in

Original Imageyi

xi yi


Bayesian Image Analysis

Eji

jiVi

ii xxyx

yY

xXxXyYyYxX

},{

222 2

1

2

1exp

Pr

PrPrPr

x

g

xX Pr xXyY

Pr y

Original Image

Degraded Image

Prior Probability

Posterior Probability

Degradation Process

Image Processing is reduced to computations of avereages, variance at each pixel and covariances of each pair of neghbouring pixels


Statistical Estimation of Hyperparameters

z

z

zXzXyY

yYzXyY

}|Pr{},|Pr{

},|,Pr{},|Pr{

},|Pr{max arg)ˆ,ˆ(

,

yY

x

g

Marginalized with respect to X

}|Pr{ xX

},|Pr{ xXyY

yOriginal Image

Marginal Likelihood

Degraded ImageV

},|Pr{ yY

Hyperparameters are determinedso as to maximize the marginal likelihood Pr{Y=y|,} with respect to ,

EM (Expectation Maximization) Algorithm


ContentsContents




Gaussian Graphical Model(Gauss Markov Random Fields)

xxyx

xxyx

yxP

Ejiji

Viii

CT22

},{

222

2

1

2

1exp

2

1

2

1exp

,,|

yyyP

V

CI

C

CI

C2

T2 2

1exp

det2

det,

yxdyxPxx 12)|(ˆ

CI

Multidimensional Gauss Integral Formulas

,max argˆ,ˆ,

gP

Maximum Likelihood Estimation EM Algorithm

),( ix

otherwise,0

},{,1

,4

Eji

Vji

ji C


One-Dimensional Signal Processing

EM Algorithm

i

i

i

0 127 255

0 127 255

0 127 255

100

0

200

100

0

200

100

0

200

ix

iy

ix̂

Original Signal

Degraded Signal

Estimated Signal

40

.,,maxarg

1,1

,ttQ

tt


Bayesian Image Analysis by Gaussian Graphical Model

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100 t

ty

f̂

ytttx 12 ))()(()(ˆ CI

Iteration Procedure of EM algorithm in Gaussian Graphical Model

EM

f̂

y

),|(max arg)ˆ,ˆ(

,

gP

40

.,,,maxarg1,1,

yttQtt

0007130ˆ

624.37ˆ

.


Image Restoration by Gaussian Graphical Model and Conventional Filters

2ˆ||

1MSE

Viii ff

V

MSE

Gaussian Graphical Model 315

Lowpass Filter(3x3) 388

(5x5) 413

Median Filter(3x3) 486

(5x5) 445

(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) MedianGaussian Gaussian Graphical Graphical

ModelModel

Original ImageOriginal ImageDegraded Degraded Image (Image (=40)=40)

V:Set of all the pixels


ContentsContents




What is an important point in What is an important point in computational complexity? computational complexity?

How should we treat the calculation of the summation over 2N configuration?

1

0

1

0

1

021

1 2

,,,x x x

N

N

xxxf

}

}

}

;,,,

1){or 0for(

1){or 0for(

1){or 0for(

;0

21

2

1

N

N

xxxfaa

x

x

x

a

N fold loops

If it takes 1 second in the case of N=10, it takes 17 minutes in N=20, 12 days in N=30 and 34 years in N=40.

Markov Chain Monte Carlo MethodBelief Propagation Method

This Talk

24 November, 2011


Strategy of Approximate Algorithm in Probabilistic Information Processing

It is very hard to compute marginal probabilities exactly except some tractable cases.

What is the tractable cases in which marginal probabilities can be computed exactly?

Is it possible to use such algorithms for tractable cases to compute marginal probabilities in intractable cases?

2 3 4

,,,,, 432111x x x x

N

N

xxxxxPxP

3 4

,,,,,, 43212112x x x

NN

xxxxxPxxP

24 November, 2011


Graphical Representations of Tractable Models



a b c d ea b c d e

a ba b c d e

b c d eX

a b

b c d e a

b c d e

a b

a b c d eb c d e



b c d e

c d eX

c d e b

c d e

b c

a b c d eb c d e

a b c

a b c

b c d ec d e

X



a b c d eb c d e

b c d ec d e

a b c d ea b c d e

c d ed e

d ee



a

b

eca b c d e f

d

f

a

b

ec

c d e f

d

fa

b

ec

b c d e f

d

f

ecd e f

d

f

ece f

d

f

ef f


Belief Propagation for Probabilistic Model on Square Grid Graph

E: Set of all the links

Eji

jijiL xxxxxPxP},{

},{21 ,,,,



1 3 4x x x xN

2 2

1 3 4

,,,,, 432122x x x x

NN

xxxxxPxP



3 4x x xN

3 4

,,,,,, 43212112x x x

NN

xxxxxPxxP

1 2 1 2


Belief Propagation

1

211222 ,x

xxPxP

Message Update Rule

3

21

5

41x

1 2

33 88

144

5

2

6

77

88

1 2

6

77


Belief Propagation on Graph with Cycles

MM

Simultaneous Fixed Point Equations of Messages

3

4 1 2

5

Average, variances and covariances can be expressed Average, variances and covariances can be expressed in terms of messages.in terms of messages.

13M

14M15M

21M3

21

5

41x

1 2


Fixed Point Equation and Iterative Method

Fixed Point Equation ** MM

Iterative Method

23

12

01

MM

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

2M


Fundamental Structures of Belief Propagation in Probabilistic Image Processing

Three Inputs and One Output Message Passing Rules

MM


Belief Propagation and EM Algorithm

Input

Output

BP EM

Update Rule of BP

２１

３

４

５


Maximization of Marginal Likelihood by EM Algorithm

.,,,maxarg1,1,

gttQtt

y

ymx

,ˆ,ˆˆ

0

0.0002

0.0004

0.0006

0.0008

0.001

0 20 40 60 80 100

Loopy Belief Propagation

Exact

0006000ˆ

335.36ˆ

.

LBP

LBP

0007130ˆ

624.37ˆ

.

Exact

Exact

24 November, 2011 39National Tsin Hua University, Taiwan

Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model

Original ImageOriginal Image

MSE:315MSE:315

MSE: 545MSE: 545 MSE: 447MSE: 447MSE: 411MSE: 411

MSE: 1512MSE: 1512

Degraded ImageDegraded Image

Lowpass FilterLowpass Filter Median FilterMedian Filter

Exact

Wiener Filter

2ˆ|V|

1MSE

Viii xx

Belief PropagationBelief Propagation

MSE:325MSE:325


Digital Images Inpainting based on MRF

Inpu

t

Ou

tpu

t

MarkovRandomFields

M. Yasuda, J. Ohkubo and K. Tanaka: Proceedings ofCIMCA&IAWTIC2005.


ContentsContents




Belief Propagation for Bayesian Networks


Factor Graph Representations of Bayesian Networks and Belief Propagations


Error Correcting Code

Y. Kabashima and D. Saad: J. Phys. A, vol.37, 2004.

High Performance Decoding Algorithm

010 000001111100000 001001011100001

0 1 0

code

010

error

decode

Parity Check Code

Turbo Code, Low Density Parity Check (LDPC) Code

majority ruleError Correcting Codes

24 November, 2011


Error Correcting Codes and Belief Propagation

)2 (mod

)2 (mod

)2 (mod

6439

5328

3217

XXXX

XXXX

XXXX

1

1

0

0

1

1

6

5

4

3

2

1

x

x

x

x

x

x

1

0

0

1

1

0

0

1

1

9

8

7

6

5

4

3

2

1

x

x

x

x

x

x

x

x

x

1

0

1

1

1

0

1

1

1

9

8

7

6

5

4

3

2

1

y

y

y

y

y

y

y

y

y

1)2 (mod 100

0)2 (mod 101

0)2 (mod 011

9

8

7

X

X

X

1 1

0

p1

1

0

0

p1

p

p

Received Word

Code Word

Binary Symmetric Channel


Error Correcting Codes and Belief Propagation

)2 (mod

)2 (mod

)2 (mod

5329

6438

4217

XXXX

XXXX

XXXX

Fundamental Concept for Turbo Codes and LDPC CodesFundamental Concept for Turbo Codes and LDPC Codes


Satisfactory Problem (3-SAT)

985872

642653431

XXXXXX

XXXXXXXXX

),,(),,(

),,(),,(),,(1

},,Pr{

985}9,8,5{872}8,7,2{

642}6,4,2{653}6,5,3{431}4,3,1{6611

xxxfxxxf

xxxfxxxfxxxfZ

xXxX


ContentsContents

1.1. IntroductionIntroduction2.2. Bayesian StatisticsBayesian Statistics3.3. Probabilistic Image ProcessingProbabilistic Image Processing4.4. Gaussian Graphical ModelGaussian Graphical Model5.5. Belief PropagationBelief Propagation6.6. Statistical Performance AnalysisStatistical Performance Analysis7.7. Various Applications of Probabilistic Various Applications of Probabilistic



SummarySummary

Fundamental Structures of Bayesian modeling Fundamental Structures of Bayesian modeling have been introduced.have been introduced.Formulation of probabilistic image processing Formulation of probabilistic image processing algorithms by means of loopy belief propagation algorithms by means of loopy belief propagation has been summarized.has been summarized.Various applications of Bayesian Network Systems Various applications of Bayesian Network Systems have been reviewed.have been reviewed.


ReferenceReferencess1. K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM

Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.

2. M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.

3. K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008

4. K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009

5. M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009.

6. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010.


TextbooksTextbooks

Kazuyuki Tanaka: Introduction of Image Processing Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese).Ltd., 2006 (in Japanese).

Kazuyuki Tanaka: Mathematics of Statistical Kazuyuki Tanaka: Mathematics of Statistical Inference by Bayesian Network, Corona Publishing Inference by Bayesian Network, Corona Publishing Co., Ltd., 2009 (in Japanese).Co., Ltd., 2009 (in Japanese).

mathematical structures of belief propagation algorithms in probabilistic information processing

Documents

dnational tsin hua university

tohoku university

joint probability pr

ajoint probability of

pair of events

computational model

possible events

marginal probabilitylet