cluster variation method and probabilistic image processing -- loopy belief propagation --
DESCRIPTION
Cluster Variation Method and Probabilistic Image Processing -- Loopy Belief Propagation --. Kazuyuki Tanaka Graduate School of Information Sciences Tohoku University [email protected] http://www.statp.is.tohoku.ac.jp/~kazu/index-e.html. Noise. - PowerPoint PPT PresentationTRANSCRIPT
28 February, 2003 University of Glasgow 1
Cluster Variation Method and Probabilistic Image Processing
-- Loopy Belief Propagation --
Kazuyuki TanakaGraduate School of Information Sciences
Tohoku [email protected]
http://www.statp.is.tohoku.ac.jp/~kazu/index-e.html
28 February, 2003 University of Glasgow 2
Probabilistic Model and Image Restoration
Original Image Degraded Image
Transmission
Noise
Image Degraded
Image OriginalImage OriginalImage Degraded
Image DegradedImage Original
P
PP
P
28 February, 2003 University of Glasgow 3
Image Restoration and Magnetic Material
Restored images are determined from a priori information and given data.
Ordered states are determined from interactions and external fields.
Feature detection from the data and image processing by means of filters.
Interpretation and prediction of physical property by means of physical model.
Similarity of Mathematical Structure
Regular lattice consisting of a lot of nodes.Neighbouring spin interactions and Markov random field
28 February, 2003 University of Glasgow 4
Massive Information Processing and Probabilistic Information Processing
Computational Complexity.Approximation algorithms for massive information processing by means of advanced mean-field methods.
Application of the cluster variation method (Bethe/Kikuchi method) to massive information processing
Cluster Variation Method is equivalent to a generalized loopy belief propagation for probabilistic inference in the artificial intelligence.
28 February, 2003 University of Glasgow 5
Important point in the application of cluster variation method to probabilistic image processing
Design of iterative algorithms for probabilistic inference based on cluster variation method (Computer Science).
Hyperparameter estimation (Statistics).
Cooperative phenomena in probabilistic models and probabilistic information processing (Physics).
28 February, 2003 University of Glasgow 6
Degradation Process and A Priori Probability in Binary Image Restoration
Degradation
Process
(Binary Symmetric Channel)
A Priori
Probability
1, yxf
f
f
),(
21,,
2,1,
),(
21,,
2,1,
21
exp
2
1exp
yxyxyxyxyx
yxyxyxyxyx
ffff
ffff
P
M
x
N
yyxyx fgPP
1 1,,fg fP
yxyx fgP ,,
0
2
10 p
28 February, 2003 University of Glasgow 7
A Priori Probability in Binary Image Restoration
f
f
),(
21,,
2,1,
),(
21,,
2,1,
21
exp
21
exp
yxyxyxyxyx
yxyxyxyxyx
ffff
ffff
P
2)(
50
1
.
1)(
1
1
)4(
250
1
.
28 February, 2003 University of Glasgow 8
Bayes Formula and A Posteriori Probability
f
gf
gf
g
ffggf
E
E
P
PPP
exp
exp
),(2
1,,2
,1,
2,,
)()(
)(
2
1
yx yxyxyxyx
yxyx
ffff
fgE
gf
)(maxargˆ,,,
,
gyxyxf
yx fPfyx
yxf
yxyx PfP,
)()( ,,\f
gfg
Maximization of Posterior Marginal
01
ln2
1
p
p f
gfEZ exp,
28 February, 2003 University of Glasgow 9
A Posteriori Probability in Binary Image Restoration
2
,,,
,, )(2
1exp
1yxyx
yxyxyx fg
ZfW
),( 1,1,,,
1,,1,
,
),( ,1,1,,
,1,,1
,
),(,,
)()(
),(
)()(
),(
,
1
yx yxyxyxyx
yxyxyxyx
yx yxyxyxyx
yxyxyx
yx
yxyxyx
fWfW
ffW
fWfW
ffWfW
ZP
gf
2',',
2','','
2,,','
,',',
',',
)(2
1exp
)(2
1exp)(
2
1exp
1,
yxyx
yxyxyxyxyxyx
yxyxyxyx
ff
fgfgZ
ffW
28 February, 2003 University of Glasgow 10
Kullback-Leibler divergence
0ln)(
gf
ff
f P
QQPQD
f
ff 1)(,0 QQ
,ln][
,lnln)()(]|[
][
ZQF
ZQQEQPQD
QF
ffgff
ff
0 PQDPQ gff
,ln][1][min ZPFQQFQ
f
f
f
gfEZ exp,
28 February, 2003 University of Glasgow 11
Kullback-Leibler Divergence
),( 1,1,,,
1,,1,
,
),( ,1,1,,
,1,,1
,
),(,, )()(
),(
)()(
),()(
yx yxyxyxyx
yxyxyxyx
yx yxyxyxyx
yxyxyx
yx
yxyxyx fQfQ
ffQ
fQfQ
ffQfQQ f
yxf
yxyx QfQ,
)()( ,,\f
f
ZFPQD ln
0ln)(
gf
ff
f P
QQPQD
f
ff 1)(,0 QQ
'',, ,',',
',', )(),(
yxyx ffyxyx
yxyx QffQ
\f
f
),(1,1,,,
1,,
1,,
),( ),(,1,1,,
,1,
,1,,,
]|[]|[]|[
]|[]|[]|[]|[}][{
yxyxyxyxyx
yxyx
yxyx
yx yxyxyxyxyx
yxyx
yxyxyxyx
WQDWQDWQD
WQDWQDWQDWQDQF
28 February, 2003 University of Glasgow 12
Basic Framework of Pair Approximation in Cluster Variation Method
FPQDQQ
minargminarg
ZQFPQD ln
1,1,
,1,1
,1,,
1,1,,1,
,
,,1,
,1,1,,1
,,,
,,
,,
yxyx
yxyx
fyxyx
yxyx
fyxyx
yxyx
fyxyx
yxyx
fyxyx
yxyxyxyx
ffQffQ
ffQffQfQ
QFPQDQQminargminarg
),(1,1,,,
1,,
1,,
),( ),(,1,1,,
,1,
,1,,,
]|[]|[]|[
]|[]|[]|[]|[}][{
yxyxyxyxyx
yxyx
yxyx
yx yxyxyxyxyx
yxyx
yxyxyxyx
WQDWQDWQD
WQDWQDWQDWQDQF
yx yxyx yxyx f fyxyx
yxyx
f fyxyx
yxyx
fyxyx ffQffQfQ
, 1,, ,1,
1,, 1,,1,
,,1,,1
,,,
28 February, 2003 University of Glasgow 13
Propagation Rule of Pair Approximation in Cluster Variation Method
yxfyx
yxyxyx
yxyxyx
yxyxyx
yxyxyxyx
yxyxyxyx
yxyxyx
yxyxyx
yxyxyxyx
yxyx fffffW
fffffWfQ
,
,1,
,,1,
,,,1
,,,1
,,,
,1,
,,1,
,,,1
,,,1
,,,,,
yx yx
yx
f fyx
yxyxyx
yxyxyx
yxyxyxyx
yxyx
fyx
yxyxyx
yxyxyx
yxyxyxyx
yxyx
yxyxyx fffffW
fffffW
f
,1 ,
,
,1,
,,1,
,,,1
,,1,,1
,
,1,
,,1,
,,,1
,,1,,1
,
,1,
,1 ,
,
yx yxf f yxyxyxyx
yxyxyx
yxyx
yxyxyxyx
yxyxyx
yxyxyxyx
yxyx
yxyxyxyx
yxyxyx
yxyx
yxyxyxyx
yxyxyx
yxyxyxyx
yxyx
yxyxyx
yx
fff
fffffW
fff
fffffW
ffQ
, ,1 ,11,1
,1,11,1
,1,1,2
,1
,1,
,,1,
,,,1
,,1,,1
,
,11,1
,1,11,1
,1,1,2
,1
,1,
,,1,
,,,1
,,1,,1
,
,1,,1
,
,
,
,
Update Rule is reduced to Loopy Belief Propagation
28 February, 2003 University of Glasgow 14
One-Body Marginal Probability of Pair Approximation in CVM
),( yx
)1,( yx
),1( yx ),1( yx
)1,( yx
)( ,1,
, yxyxyx f
)( ,,1
, yxyx
yx f
)( ,1,
, yxyxyx f
)( ,,1
, yxyx
yx f
yxfyx
yxyxyx
yxyxyx
yxyxyx
yxyxyxyx
yxyxyxyx
yxyxyx
yxyxyx
yxyxyxyx
yxyx fffffW
fffffWfQ
,
,1,
,,1,
,,,1
,,,1
,,,
,1,
,,1,
,,,1
,,,1
,,,,,
28 February, 2003 University of Glasgow 15
Two-Body Marginal Probability of Pair Approximation in CVM
)1,1( yx
),( yx
)1,( yx
),1( yx ),1( yx
)1,( yx
)( ,1,
, yxyxyx f
)( ,,1
, yxyx
yx f
)( ,1,
, yxyxyx f
)( ,11,1
,1 yxyxyx f
)1,1( yx
)( ,11,1
,1 yxyxyx f
),2( yx )( ,1
,2,1 yxyxyx f
yx yxf f yxyxyxyx
yxyxyx
yxyx
yxyxyxyx
yxyxyx
yxyxyxyx
yxyx
yxyxyxyx
yxyxyx
yxyx
yxyxyxyx
yxyxyx
yxyxyxyx
yxyx
yxyxyx
yx
fff
fffffW
fff
fffffW
ffQ
, ,1 ,11,1
,1,11,1
,1,1,2
,1
,1,
,,1,
,,,1
,,1,,1
,
,11,1
,1,11,1
,1,1,2
,1
,1,
,,1,
,,,1
,,1,,1
,
,1,,1
,
,
,
,
28 February, 2003 University of Glasgow 16
Message Propagation Rule of Pair Approximation in CVM
),( yx
)1,( yx
),1( yx ),1( yx
)1,( yx
)( ,1,
, yxyxyx f
)( ,,1
, yxyx
yx f
)( ,1,
, yxyxyx f
)( ,1,
,1 yxyxyx f
yx yx
yx
f fyx
yxyxyx
yxyxyx
yxyx
yxyx
yxyxyx
yx
fyx
yxyxyx
yxyxyx
yxyx
yxyx
yxyxyx
yx
yxyxyx
ffffW
ffW
ffffW
ffW
f
, ,1
,
,1,
,,1,
,,,1
,,1,1
,1,,1
,
,1,
,,1,
,,,1
,,1,1
,1,,1
,
,1,
,1,
,
28 February, 2003 University of Glasgow 17
Binary Image Restoration
Original images are generated by Monte Carlo simulations in the a priori probability.
Original Image Degraded Image (p=0.2) Restored Image
2)(
50
1
.
1)(
1
1
)4(
25.01
28 February, 2003 University of Glasgow 18
Binary Image Restoration
Original Image Degraded Image Restored Image
28 February, 2003 University of Glasgow 19
Hyperparameter Estimation
Maximization of Marginal Likelihood
0,,0
,,|,|,,|
ZZ
ZPPPP
ff
ffgfgg
,ˆ,ˆ,
gPmax arg
f
gfEZ exp,
M
x
N
y yxyxyxyx
yxyx
ffff
fgE
1 1 1,,,1,
,,
gf
p
p1ln
2
1
28 February, 2003 University of Glasgow 20
Binary Image RestorationOriginal images are generated by Monte Carlo simulations in the a priori probability.
Original ImageDegraded Image
(p=0.2) Restored Image
2)(
50
1
.
1)(
1
1
)4(
25.01
0.151ˆ
0.431ˆ
p
0.160ˆ
0.408ˆ
p
0.257ˆ
0.364ˆ
p
28 February, 2003 University of Glasgow 21
Binary Image Restoration
Original Image Degraded ImageMean Field
ApproximationPair Approximation
in CVM
Hyperparameters are determined so as to maximize the marginal likelihood.
28 February, 2003 University of Glasgow 22
Multi-Valued Image Restoration
Degradation Process
),(
,,yx
yxyx fgPP fg
)()1(1
)(
,,
,,
,,yxyx
yxyx
yxyx fgpQ
fgpfgP
1,,1,0, Qf yx
28 February, 2003 University of Glasgow 23
A Priori Probability in Multi-Valued Image Restoration
baba ,,
1,,1,0, Qf yx
f
f
),(
21,,
2,1,
),(
21,,
2,1,
exp
exp
yxyxyxyxyx
yxyxyxyxyx
ffff
ffff
P
Q-state Ising Model
f
f
),(1,,,1,
),(1,,,1,
,,exp
,,exp
yxyxyxyxyx
yxyxyxyxyx
ffff
ffff
P
Q-state Potts Model
Kronecker Delta
0
28 February, 2003 University of Glasgow 24
Multi-Valued Image Restoration (Q=4)Hyperparameters are determined so as to maximize the marginal
likelihood.
Degraded Image Restored
Image
Original Image
4-state Ising Model
4-state Potts Model
75.0
25.1
27153.0ˆ3
0.62021ˆ
p
0.25871ˆ3
1.10223ˆ
p
3.03 p
28 February, 2003 University of Glasgow 25
Multi-Valued Image Restoration (Q=4)Hyperparameters are determined so as to
maximize the marginal likelihood.
Degraded Image（ 3p=0.3
）
4-state Potts Model
Original Image4-state Ising
Model
28 February, 2003 University of Glasgow 26
Summary
Probabilistic Image Processing by Bayes Formula
Cluster Variation Method and Loopy Belief Propagation
Binary Image Restoration
Multi-Valued Image Restoration
28 February, 2003 University of Glasgow 27
Other Practical Applications
Edge DetectionSegmentationTexture AnalysisImage CompressionMotion DetectionColor Image
28 February, 2003 University of Glasgow 28
Other Theoretical Works
Hyperparameter Estimation by EM algorithmStatistical Performance Estimation and Spin Glass Theory
Replica methodInequality of Statistical Quantity
Line FieldGeneralized Loopy Belief Propagation and Cluster Variation MethodInformation Geometry and Cluster Variation Method