physical fluctuomatics 7th~10th belief propagation appendix

Physics Fluctuomatics (Tohoku University) 1

Physical Fluctuomatics7th~10th Belief propagation

Appendix

Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/


Textbooks

Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5.

ReferencesH. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011.M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.


Probabilistic Model for Ferromagnetic Materials

p p

p p

)1,1()1,1()1.1()1.1( PPPP

pPP )1.1()1,1(

11 a

1

12 a

1

11

1 1

p

PP

21

)1.1()1,1(


Probabilistic Model for Ferromagnetic Materials

Prior probability prefers to the configuration with the least number of red lines.

> >=

Lines Red of #Lines Blue of # )21()( ppaP

p p

11 a 112 a 111 1 1


More is different in Probabilistic Model for Ferromagnetic Materials

Disordered State

Ordered State

Sampling by Markov Chain Monte Carlo method

p p

Small p Large p

p p

More is different.

p21 p

21

Critical Point(Large fluctuation)


Fundamental Probabilistic Models for Magnetic Materials

Since h is positive, the probablity of up spin is larger than the one of down spin ．

1)exp(

)exp()(

aha

haaP1a

+1 1

he he

)tanh()(1

haaPma

h ： External Field

)(tanh1)()( 2

1

2 haPmaaVa

Variance

Average

0h



Since J is positive, (a1,a2)=(+1,+1) and (1,1) have the largest probability ．

1 121

2121

1 2

)exp()exp(

),(

a aaJa

aJaaaP

11 a

0),(1 1

21111 2

a a

aaPam

J ： Interaction

1),()(1 1

212

1111 2

a a

aaPmaaVVariance

Average

0J

Je Je

+1 +1 1 1

+1 +1 11

12 aJe Je



a

aEZ

))(exp(

Eji

jiVi

i aaJahaE},{

)(

Translational Symmetry

),( EVJJ

h h

)(exp1)( aEZ

aP

),,,( 21 Naaaa

E ： Set of All the neighbouring Pairs of Nodes

1ia 1ia

N

i ai aPa

Nm

1)(1

Problem: Compute

)'()()'()( aPaPaEaE



Eji

jiVi

i aaJahaE},{

)(

N

i ai

NhaPa

Nm

10)(1limlim

)(exp1)( aEZ

aP

),,,( 21 Naaaa

1ia

Problem: Compute


),( EV

J

J

h h

Spontaneous Magnetization


Mean Field Approximation for Ising Model

)},{( 0))(( Ejimama ji We assume that the probability for configurations satisfying

Vi

iaJmhaE )4()(

2mmamaaa ijji

Eji

jiVi

i aaJahaE},{

iJm

Jm

JmJm

h

are large.


Mean Field Approximation for Ising Model

)4tanh()(1

1JmhaPa

Nm

N

i ai

Vi

ii aPaEZ

aP )())(exp(1)(

Fixed Point Equation of m)(mm

We assume that all random variables ai are independent of each other, approximately.

Vi

iaJmhaE )4()(


Fixed Point Equation and Iterative Method

• Fixed Point Equation ** MM



• Fixed Point Equation ** MM • Iterative Method

0

xy

)(xy

y

x*M




0M0

xy

)(xy

y

x*M




01 MM

0M

1M

0

xy

)(xy

y

x*M




12

01

MMMM

0M1M

1M

0

xy

)(xy

y

x*M




12

01

MMMM

0M1M

1M

0

xy

)(xy

y

x*M

2M




23

12

01

MMMMMM

0M1M

1M

0

xy

)(xy

y

x*M

2M


Marginal Probability Distribution in Mean Field Approximation

))4exp((1

)()(1 2 1 1

ii

a a a a aii

aJmhZ

aPaPi i N

iJm

Jm

JmJm

h

1

)(ia

iii aPam

))4tanh(( mJhm Jm： Mean Field


Advanced Mean Field Method

))4exp((1)( ii

ii ahZ

aP

)))(3exp((1),( jijii

jiij aJaaahZ

aaP

h

h

h

1

),()(ja

jiijii aaPaP

))3tanh()(tanh(arctanh hJ

Bethe Approximation

Kikuchi Method (Cluster Variation Meth)

： Effective Field

Fixed Point Equation for

J


Average of Ising Model on Square Grid Graph

(a) Mean Field Approximation(b) Bethe Approximation(c) Kikuchi Method (Cluster Variation Method)(d) Exact Solution （ L. Onsager ， C.N.Yang ）

J/1

a

iNh

aPa

)(limlim0

Ejiji

Vii aaJah

ZaP

},{

exp1 ),( EVJJ

h h


Model Representation in Statistical Physics

),,,(},,,Pr{ 212211 NNN aaaPaAaAaA

a

aEZ

))(exp(

)(}Pr{ aPaA

))(exp(1)( aEZ

aP

),,,( 21 NAAAA

Gibbs Distribution Partition Function

)))(exp(ln(ln a

aEZF

Free Energy

Energy Function


Gibbs Distribution and Free Energy

Gibbs Distribution

ZPFaQQFaQ

ln][}1)(|][{min

))(exp(1)( aEZ

aP

)(ln)()()(][ aQaQaQaEQFaa

Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a)

Free Energy Functional of Trial Probability Distribution Q(a)

a

aEZ

))(exp(lnlnFree Energy


Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional

ZPFaQQFaQ

ln][}1)(|][{min

)(

)(exp)(exp)(ˆ aPaEaEaQ

a

1)()())(ln)((1)(

aaaaQaQaQaEaQQFQL

01)(ln)()(

aQaEaQQL

1)(exp)(ˆ aEaQ

Normalization Condition


Kullback-Leibler Divergence and Free Energy

0)()(ln)(

aP

aQaQPQDa

aaQaQ

1)( ,0)(

ZQF

ZaQaQaEaQPQD

QFaa

ln][

ln)(ln)()()(]|[

][

0)()( PQDaPaQ

))(exp(1)( aEZ

aP

}1)(|]|[{minarg}1)(|][{minarg aQaQ

aQPQDaQQF


Interpretation of Mean Field Approximation as Information Theory

Vi

ii aQaQ )()(

)(

)(ln)(aPaQaQPQD

a

))(exp(1 aEZ

aP and

Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P]

1 2 1 1 2

)()()(\ a a a a a aaa

iii i i Ni

aQaQaQ

Minimization of Kullback-Leibler Divergence between


Interpretation of Mean Field Approximation as Information Theory

Eji

jiVi

i aaJahaE},{

)(

Vi a

iiVi a

ii

aPaV

aPaV

m1

)(||

1)(||

1

)(exp1)( aEZ

aP

),,,( ||21 Vaaaa

1ia

Problem: Compute


),( EV

J

J

h h

Magnetization

1 2 1 1 2

)()()(\ a a a a a aaa

iii i i Ni

aPaPaP


Kullback-Leibler Divergence in Mean Field Approximation for Ising Model

Vi

ii aQaQ )()(

ZViQFPQD i ln|MF

)(

)(ln)(aPaQaQPQD

a

Viii

Ejiji

Viii

QQQQJ

QhViQF

1},{ 11

1MF

ln))()()((

)(}]|[{

1 2 1 1 2

)(

)()(\

a a a a a a

aaii

i i i N

i

aQ

aQaQ

Eji

jiVi

i aaJahaE},{

)( )(exp1)( aE

ZaP


Minimization of Kullback-Leibler Divergence and Mean Field Equation

)( ))(ˆ(exp1ˆ1

ViQJhZ

Qij

ji

i

} ,1)(|]|[{minarg)}(ˆ{}{

ViQPQDQ iQii

Fixed Point Equations for {Qi|iV}

Variation

1 1

))(ˆ(exp

ij

ji QJhZi

Set of all the neighbouring nodes of the node i

Ejiji },{


Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model

iiii amaQ21

21)(

1

)(ia

iiia

ii aQaaQam

),,,( 21 Naaaa 1ia

ia

iiia

iia

iii

aii

ai

aii

iiii

maQadddacaaQa

aQccdacaQ

adacaQ

iii

iii

21)(

212)()(

21)(

212)()(

1)( )(

111

111

2

1 2 1 1

)()()(\ a a a a aaa

iii i Ni

aQaQaQ


Conventional Mean Field Equation in Ising Model

)4tanh( Jmhm

iiiiiaa

ii maamaQaPaPi

21

21

21

21)(ˆ)()(

\

maPaN

N

i ai

1)(1

Fixed Point Equation

mmmm N 21

Eji

jiVi

i aaJahaE},{

)(

))4exp((1))(ˆ(exp1)(ˆ1

ii

iij

ji

ii aJmhZ

aQJhZ

aQ

VJ

J


h h

)( 4|| Vii


Interpretation of Bethe Approximation (1)

Eji

jiVi

i aaJahaE},{

)(

)(exp1)( aEZ

aP

),,,( ||21 Vaaaa

1ia


),( EV

J

J

h h

1 2 1 1 2

)()()(\ a a a a a aaa

iii i i Ni

aPaPaP

1 2 1 1 2 1 1 2

)()(),(},\{ a a a a a a a a aaaa

jiiji i i j j j Nji

aPaPaaP

Eji

jiij aaZ

aP},{

),(1)(

jijijiij aJaha

jha

iaa

||1

||1exp),(

a Eji

jiij aaZ

},{

),(

Compute

and


ZQFPQD ln

aQaQaaWaaQ

aQaQaaWaQ

aQaQaaWaQQF

aEji a ajiijjiij

aEji a ajiij

aaa

aEjijiij

a

i j

i j ji

ln)(,ln,

ln)(,ln)(

ln)(,ln)(

},{

},{ ,\

},{

0ln)(

aP

aQaQPQDa

Free EnergyKL Divergence

Eji

jiij aaWZ

P},{

,1x

ji aaa

jiij

aQ

aaQ

,

)(

),(

\

33Physics Fluctuomatics (Tohoku

University)


ZQFPQD ln

Ejijjiiijij

Viii

Ejiijij

a

Ejiijij

QQQQQQ

QQ

WQ

aQaQ

WQQF

},{

},{

},{

lnln,ln,

ln

,ln,

ln)(

,ln,

Bethe Free Energy

Free EnergyKL Divergence

Eji

jiij aaWZ

aP},{

,1

ji aaajiij aQaaQ

,

)(),(\

iaaii aQaQ

\

)()(


University)


FPQDQQ

minargminarg

,iji QQ

ZQQFPQD iji ln,Bethe

ijiQQQ

QQFPQDiji

,minargminarg Bethe,

1,

iji QQ

Ejijjiiijij

Viii

Ejiijijiji

QQQQQQ

QQWQQQF

},{

},{Bethe

lnln,ln,

ln,ln,,


University)


Ejiijij

Viii

Vi ijijijii

ijiiji

QQ

QQ

QQFQQL

},{

},{,

BetheBethe

1,1

,

,,

1, ,,,minarg Bethe,

ijiijiijiQQ

QQQQQQFiji

Lagrange Multipliers to ensure the constraints


University)


Ejiijij

Viii

Vi ijijijji

Ejijjiiijij

Viii

Ejiijij

Ejiijij

Viii

Vi ijijijiiijiiji

QQQQ

QQQQQQ

QQQ

QQ

QQQQFQQL

},{},{,

},{

},{

},{

},{,BetheBethe

1,1,

lnln,ln,

ln,ln,

1,1

,,,

0,Bethe

iji

ii

QQLxQ

• Extremum Condition

0,, Bethe

ijijiij

QQLxxQ


University)


FGfP yxyxyx g ,,,

ik

ikiiii ai

aQ )(1||

1exp },{, )()(exp,, 2}2,1{,21}2,1{,121122112 aaaaaaQ

Extremum Condition 0,Bethe

ijiii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ


University)

115114

1131121

111

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaaW

aMaMaMZ

aaQ

)()(exp\

},{, ijik

ikijii aMa


FGfP yxyxyx g ,,,14 2

5

13M

14M

15M

12M

3

115114

1131121

111

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaaW

aMaMaMZ

aaQ

Extremum Condition 0,Bethe

ijiii

QQLxQ 0,

, Bethe

iji

jiij

QQLxxQ


University)

26M

14

5

13M

14M

15M

12W3

2

6

27M

8

7

28M


FGfP yxyxyx g ,,,14 2

5

13M

14M

15M

12M

3

412W

1

5

13M

14M

15M

3

26M2

6

27M

8

7

28M

,121 QQ

115114

1131121

111

aMaM

aMaMZ

aQ

2282272262112

11511411312

2112

,

1,

aMaMaMaa

aMaMaMZ

aaQ

1514

1312

21

,

MM

M

M

Message Update Rule


University)


15141312

15141312

21 ,

,

MMMW

MMMW

M

1

3

4 2

5

13M

14M

15M

21M

14

5

3

2

6

8

7

2a

14 2

5

3

=

Message Passing Rule of Belief Propagation

It corresponds to Bethe approximation in the statistical mechanics.


University)


1 1 \

1 \,

,

jikikij

jikikij

ji MW

MW

M


University)

))tanh()(tanh(arctanh\

jik

ikji hJ

jijiM exp

))3tanh()(tanh(arctanh hJ

ji Translational Symmetry


Summary

Statistical Physics and Information TheoryProbabilistic Model of FerromagnetismMean Field TheoryGibbs Distribution and Free EnergyFree Energy and Kullback-Leibler DivergenceInterpretation of Mean Field Approximation as Information TheoryInterpretation of Bethe Approximation as Information Theory

physical fluctuomatics 7th~10th belief propagation appendix

Documents

magnetic materials problem

oxford university press

information processing

ijmjmjmjmhare large

critical phenomena

stateordered statesampling

elements of phase transitions

number of red lines