the factor graph approach to model-based signal processing
DESCRIPTION
The Factor Graph Approach to Model-Based Signal Processing. Hans-Andrea Loeliger. Outline. Introduction Factor graphs Gaussian message passing in linear models Beyond Gaussians Conclusion. Outline. Introduction Factor graphs Gaussian message passing in linear models Beyond Gaussians - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/1.jpg)
The Factor Graph Approach to Model-Based Signal Processing
Hans-Andrea Loeliger
![Page 2: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/2.jpg)
2
Outline
Introduction
Factor graphs
Gaussian message passing in linear models
Beyond Gaussians
Conclusion
![Page 3: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/3.jpg)
3
Outline
Introduction
Factor graphs
Gaussian message passing in linear models
Beyond Gaussians
Conclusion
![Page 4: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/4.jpg)
4
Introduction
Engineers like graphical notation
It allow to compose a wealth of nontrivial algorithms from tabulated “local” computational primitive
![Page 5: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/5.jpg)
5
Outline
Introduction
Factor graphs
Gaussian message passing in linear models
Beyond Gaussians
Conclusion
![Page 6: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/6.jpg)
6
Factor Graphs
A factor graph represents the factorization of a function of several variables
Using Forney-style factor graphs
![Page 7: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/7.jpg)
7
Factor Graphs cont’d
Example:
1 2 3 4( , , , , ) ( ) ( , , ) ( , , ) ( )f u w x y z f u f u w x f x y z f z
![Page 8: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/8.jpg)
8
Factor Graphs cont’d
1 2 3 4 5( , , , , ) ( ) ( ) ( | , ) ( | ) ( | )f u w x y z f u f w f x u w f y x f z x
(a) Forney-style factor graph (FFG); (b) factor graph as in [3]; (c) Bayesian network; (d) Markov random field (MRF)
![Page 9: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/9.jpg)
9
Factor Graphs cont’d
Advantages of FFGs:
suited for hierarchical modeling
compatible with standard block diagram
simplest formulation of the summary-product message update rule
natural setting for Forney’s result on FT and duality
![Page 10: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/10.jpg)
10
Auxiliary Variables
Let Y1 and Y2 be two independent observations of X:
1 2 1 2( , , ) ( ) ( | ) ( | )f x y y f x f y x f y x
1 2 1 2( , ', ", , ) ( ) ( | ') ( | ") ( , ', ")f x x x y y f x f y x f y x f x x x
( , ', ") ( ') ( ")f x x x x x x x 1 2 1 2
' "
( , , ) ( , ', ", , ) ' "x x
f x y y f x x x y y dx dx
![Page 11: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/11.jpg)
11
Modularity and Special Symbols
Let and with Z1, Z2 and X independent
The “+”-nodes represent the factors and
1 1Y X Z 2 2Y X Z
1 1( )x z y 2 2( )x z y
![Page 12: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/12.jpg)
12
Outline
Introduction
Factor graphs
Gaussian message passing in linear models
Beyond Gaussians
Conclusion
![Page 13: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/13.jpg)
13
Computing Marginals
Assume we wish to compute
For example, assume that can be written as
1
1,...,
( ) ( ,..., )n
k
k k nx xexcept x
f x f x x
1 7( ,..., )f x x
1 7 1 1 2 2 3 1 2 3 4 4
5 3 4 5 6 5 6 7 7 7
( ,..., ) ( ) ( ) ( , , ) ( )
( , , ) ( , , ) ( )
f x x f x f x f x x x f x
f x x x f x x x f x
![Page 14: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/14.jpg)
14
Computing Marginals cont’d
3 3 3 3( ) ( ) ( )C Df x x x
1 2
3 1 1 2 2 3 1 2 3,
( ) ( ) ( ) ( , , )Cx x
x f x f x f x x x
4 5
3 4 4 5 3 4 5 5,
( ) ( ) ( , , ) ( )D Fx x
x f x f x x x x
6 7
5 6 5 6 7 7 7,
( ) ( , , ) ( )Fx x
x f x x x f x
1 2
3 1 2 3 1 2 3,
( ) ( ) ( ) ( , , )C A Bx x
x x x f x x x
4 5
3 5 3 4 5 4 5,
( ) ( , , ) ( ) ( )D E Fx x
x f x x x x x
6 7
5 6 5 6 7 7,
( ) ( , , ) ( )F Gx x
x f x x x x
![Page 15: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/15.jpg)
15
Message Passing View cont’d
1 2
3
4 5 6 7
5
3
3 3 1 2 3 1 2 3,
5 3 4 5 4 6 5 6 7 7, ,
( ) ( ) ( ) ( , , )
( , , ) ( ) ( , , ) ( )
X
X
X
A Bx x
E Gx x x x
f x x x f x x x
f x x x x f x x x x
��������������
�
�
![Page 16: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/16.jpg)
16
Sum-Product Rule
The message out of some node/factor along some edge is formed as the product of and all incoming messages along all edges except , summed over all involved variables except
kX
lf
kX
kX
3 33 3 3 3( ) ( ) ( ) ����������������������������
X Xf x x x
lf
![Page 17: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/17.jpg)
17
denotes the message in the direction of the arrow
denotes the message in the opposite direction
Arrows and Notation for Messages
��������������
X
�X
![Page 18: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/18.jpg)
18
Marginals and Output Edges
'' "
'
( ) ( ') ( ") ( ") ( ') ' "
( ) ( )
X X Xx x
X X
x x x x x x x dx dx
x x
������������������������������������������
����������������������������
![Page 19: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/19.jpg)
19
Max-Product Rule
The message out of some node/factor along some edge is formed as the product of and all incoming messages along all edges except , maximized over all involved variables except
kX
lf
kX
kX
11
,...,
( ) max ( ,..., )n
k
k k nx xexcept x
f x f x x
lf
![Page 20: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/20.jpg)
20
Message of the form:
Arrow notation:
/ is parameterized by mean / and variance /
2 2( ) / 2( ) x mx e
Scalar Gaussian Message
��������������
X �X
��������������Xm
�Xm
2��������������
X2
�X
![Page 21: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/21.jpg)
21
Scalar Gaussian Computation Rules
![Page 22: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/22.jpg)
22
Vector Gaussian Messages
Message of the form:
Message is parameterized
either by mean vector m and covariance matrix V=W-1
or by W and Wm
( ) expH
x x m W x m
![Page 23: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/23.jpg)
23
Vector Gaussian Messages cont’d
Arrow notation:
is parameterized by and or by and
Marginal:
is the Gaussian with mean and covariance matrix
��������������
X Xm��������������
XV��������������
XW��������������
X XW m����������������������������
( ) ( )X Xx x ����������������������������
Xm1
X XV W
k kk
k k k kk k
X XX
X X X XX X
W W W
W m W m W m
����������������������������
��������������������������������������������������������
![Page 24: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/24.jpg)
24
Single Edge Quantities
![Page 25: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/25.jpg)
25
Elementary Nodes
![Page 26: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/26.jpg)
26
Matrix Multiplication Node
![Page 27: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/27.jpg)
27
Composite Blocks
![Page 28: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/28.jpg)
28
Reversing a Matrix Multiplication
![Page 29: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/29.jpg)
29
Combinations
![Page 30: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/30.jpg)
30
General Linear State Space Model
1K K K K K
K K K
X A X B U
Y C X
![Page 31: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/31.jpg)
31
General Linear State Space Model
If is nonsingular
and - forward
and - backward
If is singular
and - forward
and - backward
Cont’d
1kX
��������������1kY
�
kX��������������
kY�
'kX�
1'kX
�
kA
kA
1kX
��������������1kY
�
kX��������������
kU��������������
kX�
1kX
�
![Page 32: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/32.jpg)
32
General Linear State Space Model
By combining the forward version with backward version, we can get
Cont’d
k kk
k k k kk k
X XX
X X X XX X
W W W
W m W m W m
����������������������������
��������������������������������������������������������
![Page 33: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/33.jpg)
33
Gaussian to Binary
( ) ( ) ( ) ( )X Y Y
y
x y x y dy x ������������������������������������������
2
( 1) 2ln
( 1)
YXX
X Y
mL
����������������������������
�������������� ��������������
21
Y X
Y X X
m m
m
�������������� �
��������������������������� ��
![Page 34: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/34.jpg)
34
Outline
Introduction
Factor graphs
Gaussian message passing in linear models
Beyond Gaussians
Conclusion
![Page 35: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/35.jpg)
35
Message Types
A key issue with all message passing algorithms is the representation of messages for continuous variables
The following message types are widely applicable
Quantization of continuous variables
Function value and gradient
List of samples
![Page 36: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/36.jpg)
36
Message Types cont’d
All these message types, and many different message computation rules, can coexist in large system models
SD and EM are two example of message computation rules beyond the sum-product and max-product rules
![Page 37: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/37.jpg)
37
LSSM with Unknown Vector C
![Page 38: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/38.jpg)
38
Steep Descent as Message Passing
Suppose we wish to find
( ) ( ) ( )A Bf f f
max arg max ( )f
(ln ( )) (ln ( )) (ln ( ))A B
d d df f f
d d d
![Page 39: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/39.jpg)
39
Steep Descent as Message Passing
Steepest descent:
where s is a positive step-size parameter
Cont’d
ln ( ) |old
new oldd
s fd
![Page 40: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/40.jpg)
40
Steep Descent as Message Passing
Gradient messages:
Cont’d
( ) ln ( )d
d
�������������� �
![Page 41: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/41.jpg)
41
Steep Descent as Message PassingCont’d
( ) ( ) ( ) ( )
( ) ( )
X Yx y
X Yx
x x y x dxdy
x x dx
��������������������������� ��
����������������������������
![Page 42: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/42.jpg)
42
Outline
Introduction
Factor graphs
Gaussian message passing in linear models
Beyond Gaussians
Conclusion
![Page 43: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/43.jpg)
43
Conclusion
The factor graph approach to signal processing involves the following steps:
1) Choose a factor graph to represent the system model
2) Choose the message types and suitable message computation rules
3) Choose a message update schedules
![Page 44: The Factor Graph Approach to Model-Based Signal Processing](https://reader035.vdocuments.us/reader035/viewer/2022062301/5681479d550346895db4d24a/html5/thumbnails/44.jpg)
44
Reference
[1] H.-A. Loeliger, et al., “The factor graph approach to model-based signal processing”
[2] H.-A. Loeliger, “An introduction to factor graphs,” IEEE Signal Proc. Mag., Jan. 2004, pp.28-41
[3] F.R. Kschischang, B.J. Fery, and H.-A. Loeliger, “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform. Theory, vol. 47, pp.498-519