![Page 1: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/1.jpg)
Information Signal ProcessingInformation Signal ProcessingJoseph A. OJoseph A. O’’SullivanSullivan
Electronic Systems and Signals Research LaboratoryCenter for Security Technologies
Department of Electrical and Systems Engineering Washington University
[email protected]://essrl.wustl.edu/~jao
Supported by: ONR, NSF, NIH, Boeing Foundation, DARPASpecial thanks to Naveen Singla
![Page 2: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/2.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
2
CollaboratorsCollaborators
Jasenka BenacMichael D. DeVoreAndrew (Lichun) LiClayton MillerLee MontagninoRyan Murphy Natalia SchmidNaveen SinglaBrandon WestoverShenyu Yan
G. James BlaineRoger ChamberlainMark FranklinDaniel R. FuhrmannRonald S. IndeckChenyang LuPierre Moulin, UIUCMarcel MullerRobert PlessDavid G. PolitteChrysanthe PrezaAndrew Singer, UIUC Donald L. SnyderBruce R. WhitingJeffrey F. Williamson, VCULihao Xu
Faculty Current and Former Students
![Page 3: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/3.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
3
OutlineOutline
• DSP ISP- Signal processing - Time, space, samples- Information, distortion, transmission
• Graphical Models- Data models - Computational and communication models- X-ray CT imaging- Iterative decoding
• Message Passing EM Algorithms- Graphical models- Projections
• Applications Revisited• Speculation on Trends• Conclusions
![Page 4: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/4.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
4
OutlineOutline
• DSP ISP- Data models, computational models, algorithms- Central role of information
• Graphical Data Models- X-ray CT imaging- Iterative decoding
• Message Passing EM Algorithms
• Applications Revisited• Speculation on Trends• Conclusions
![Page 5: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/5.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
5
Signal Processing
Information TheoryComputation andCommunication
FFTFFT
MultiresolutionMultiresolutionanalysisanalysis
![Page 6: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/6.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
6
X(0)
X(1)
X(2)
X(3)
X(4)
X(5)
X(6)
X(7)
x(0)
x(4)
x(2)
x(6)
x(1)
x(5)
x(3)
x(7)
FFT
![Page 7: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/7.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
7
Signal Processing
Information Theory
Numerical analysisNumerical analysisProcessors: parallel, Processors: parallel,
ASIC, etc.ASIC, etc.Systolic architecturesSystolic architecturesFFTWFFTW
Computation andCommunication
![Page 8: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/8.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
8
Signal Processing
Information Theory
Numerical analysisNumerical analysisProcessors and Processors and
architecturesarchitecturesFFT, FFTWFFT, FFTWTransversal filtersTransversal filters
MRAMRA
Complexity theoryComplexity theoryGraphical modelsGraphical models
KalmanKalman filtersfiltersCompression Compression
Computation andCommunication
![Page 9: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/9.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
9
Computation andCommunication
Information Signal ProcessingInformation Signal Processing
Signal Processing
Information Theory
ComplexityComplexity--constrained processingconstrained processingSignal processing on graphsSignal processing on graphsDistributed signal processingDistributed signal processingDistributed information theoryDistributed information theoryDistributed computation and Distributed computation and
communicationcommunicationOptimal information extraction, Optimal information extraction,
communication, computationcommunication, computation
![Page 10: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/10.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
10
Distributed sensing,communication, computation
““The architecture for a The architecture for a fully netted maritime forcefully netted maritime force””
![Page 11: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/11.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
11
Wireless Sensor NetworksWireless Sensor Networks
http://www.greatduckisland.net/index.php
Great Duck Island Habitat
![Page 12: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/12.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
12
![Page 13: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/13.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
13
Information Signal ProcessingInformation Signal Processing• Measure information quantitatively
– Entropy rates, mutual information, relative entropy– Rate-distortion theory
• Graphical data models– Forward likelihood models for data– Priors, penalties, and multiresolution models– Abstractions to random graphs
• Constrained implementations– Parameterize implementations: communication rates,
computational complexity, computation time
• Principled approach to complexity-constrained signal modeling, data analysis, and algorithm development
![Page 14: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/14.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
14
OutlineOutline
• DSP ISP- Data models, computational models, algorithms- Central role of information
• Graphical Data Models- X-ray CT imaging- Iterative decoding
• Message Passing EM Algorithms
• Applications Revisited• Speculation on Trends• Conclusions
![Page 15: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/15.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
15
Graphical Data ModelsGraphical Data Models
•• Model 1: Model 1: y = Hsy = Hs– y is n × 1, s is m × 1,
and H is n × m– yj depends on sk if hjk ≠ 0– Defines a graphical model
•• Model 2:Model 2:
– Neighborhood structure– Bipartite graph model
•• Model 3:Model 3:– RVs on edges of graph
k
j∏ ∈=
jkj jksypp ))(,|()|( sy
∏ ∏∈
=k kj
kjk sxpp)(
)|()|( sx
∏ ∈=j
jkj jkxypp ))(,|()|( xy
![Page 16: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/16.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
16
Tomography
S
D
Nonrandom Graphs
Line integrals through patientQuantization point spread function
weights on edges of graphHelps organize computations
Siemens Somotom Emotion
![Page 17: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/17.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
17
Computational ModelsComputational Models
System model accounts for:• Information extraction
problem definition• Compression of sensor data• Network throughput• Processor cycles per
instruction• Size of processor local
memory• Communication bandwidth
of each link• etc.
![Page 18: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/18.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
18
Computational ModelsComputational Models
• Local resources plus remote• Communicate observation as well as classification
- Human in the loop- Remote contribution to classification when available
• Dynamic resource availability• Sequence of partial classifications (an,θn)
![Page 19: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/19.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
19
Progress: Computational Graph Progress: Computational Graph Same as Data Model GraphSame as Data Model Graph
•• Message passing algorithmsMessage passing algorithms– Pearl’s belief propagation– Iterative decoding
» Turbo-codes, parallel concatenated codes» Low density parity check codes» Repeat-accumulate codes, serial concatenated codes
– Iterative equalization and decoding
•• ExpectationExpectation--Maximization (EM) AlgorithmsMaximization (EM) Algorithms– Graphical models– General problem– Gaussian, Poisson (emission tomography, transmission
tomography)– Abstract examples on random graphs
![Page 20: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/20.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
20
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
10011010101000010101010100011010000101010101100110010100100101010010100010100010101101001010100110
H Regular (3,6) n=14
Random Graphs
Comment: LDPC parity check matrix
![Page 21: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/21.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
21
⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
001010100000100100000010010001001000000100010001100000001100001000100010010001010000000010000101
H Regular (2,3) n=12
Random Graphs
![Page 22: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/22.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
22
⎟⎟⎟⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜⎜⎜⎜
⎝
⎛
=
10011010101000010101010100011010000101010110101010010100100101010010100010100010110101001001010110
H Irregular n=14
Random Graphs
![Page 23: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/23.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
23
Tomography
S
D
![Page 24: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/24.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
24
k
j
νjk
µjk
µ
µ µ
µ
νν
),(
),()1(
')1(
)('
)1(
stateg
statefmkjk
mjk
mjkj
mjk
++
+
=
=
µν
νµ
Message Passing AlgorithmsMessage Passing Algorithms
![Page 25: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/25.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
25
TomographyS
D
![Page 26: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/26.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
26
![Page 27: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/27.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
27
Iterative Decoding Message PassingIterative Decoding Message Passing
∑∈
+ =kjk
mjk
mjk
\)('
)('
)1( νµ
∏∈
++
−=jkj
mkjz
mjk k
\)('
)1('
)1(
2tanh)1(
2tanh
µν
Codeword Bit Nodes
zk
xj
Check Nodes
jkµ
jkν
![Page 28: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/28.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
28
Iterative Decoding Message PassingIterative Decoding Message Passing
-6
-5
-4
-3
-2
-1
0
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
SNR [dB]
Bit
erro
r rat
e, L
og B
ase
10 ISI-freeBIAWGN Capacity
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
![Page 29: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/29.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
29
OutlineOutline
• DSP ISP- Data models, computational models, algorithms- Central role of information
• Graphical Data Models- X-ray CT imaging- Iterative decoding
• Message Passing EM Algorithms
• Applications Revisited• Speculation on Trends• Conclusions
![Page 30: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/30.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
30
ML Problem: { }xsxxys
dpp∫ )|()|(lnmax
EM Algorithm: xsxxy
syxsyxs
dpp∫ ⎥
⎦
⎤⎢⎣
⎡ ΦΦ
Φ )|()|(),|(ln),|(minmin
EM AlgorithmEM Algorithmsk
yj
Hidden Data
Incomplete Data
xjk
![Page 31: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/31.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
31
VariationalVariational RepresentationsRepresentations•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then
• Special Case: f is ln
• Basis for EM; see also De Pierro, Lange, Fessler
∑
∑ ∑≥=
≤
iii
i ii
irii
rr
xfrxf
0,1
)()( 1
⎭⎬⎫
⎩⎨⎧
=Φ≥ΦΦ=
ΦΦ−=⎟
⎠
⎞⎜⎝
⎛
∑
∑∑∈Φ
iii
i i
ii
ii q
q
1,0:
lnminln
P
P
![Page 32: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/32.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
32
EM AlgorithmEM Algorithm
∫=Φ +
')|'()'|()|()|(),|(
)(
)()()1(
xsxxysxxysyx
dpppp
m
mmm
xsxsyxss
dp, mmm )|(ln)|(argmax )()1()1( ∫ ++ Φ=
Assume the factorizations (Model 3):
∏ ∏∈
=k kj
kjk sxpp)(
)|()|( sx
∏ ∈=j
jkj jkxypp ))(,|()|( xy
These computations become local and thus message passing
In general, these are global computations
![Page 33: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/33.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
33
Message Passing EM AlgorithmMessage Passing EM Algorithm
∫ ∏∏∫ ∏∏
∈∈
≠∈∈+
∈
∈=Φ
)(''
)('
)('''
'),(''
)('
)('''
1(
)|())(',|(
)|())(',|(),|(
jkjk
jk
mkjkjkj
kkjkjk
jk
mkjkjkj
(m)jjk
)m
dxsxpjkxyp
dxsxpjkxypyx
KK
KK
K
Ks
jkkjkm
jjkm
s
mk dxsxp,yxs
k
)|(ln)|(argmax )()1()1( ∫ ++ Φ= s
sk
yj
Input Data
Measured Data
xjk
)( jkxΦ
ks
![Page 34: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/34.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
34
GaussianGaussian--MAPMAP
)2
,0(~
),0(~;
0 Iw
IswHsy
NN
PN+=
∑∈
+
+=
)(
)(
0
)1( ˆ
|)(|2|)(|
ˆkj
ljk
lk x
jNkP
Ps
⎟⎟⎠
⎞⎜⎜⎝
⎛−+= ∑
∈
+++
)('
)1('
)1()1( ˆ|)(|
1ˆˆjk
lkj
lk
ljk sy
jsx
sk
yj
Input Data
Measured Data
xjk
jkx
ks
![Page 35: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/35.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
35
GaussianGaussian--MAPMAP[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
![Page 36: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/36.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
36
Emission TomographyEmission Tomography
y ~ Poisson(Hλ)λ: Mean of emitted photons
∑∈
+ =)(
)()(
)1( ˆ|)(|
ˆˆkj
mj
mkm
k qk
λλ
∑∈
=
)('
)('
)(
ˆˆ
jk
mk
jmj
yq
λ
k
yj
Pixels
Measured Data
jqkλ
![Page 37: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/37.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
37
Emission TomographyEmission Tomography[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
![Page 38: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/38.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
38
Emission TomographyEmission Tomography[50000,1000] regular (3,150) matrix[50000,1000] regular (3,150) matrix
![Page 39: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/39.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
39
Transmission TomographyTransmission Tomography
y ~ Poisson(I0exp(-Hλ))λ: photon attenuation ⎥
⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−=
∑∑
∈
∈+
)(
)()()()1( ln1ˆˆ
kj
mj
kjj
mk
mk q
y
zλλ
⎥⎦
⎤⎢⎣
⎡−= ∑
∈ )(
)(0
)( ˆexpjk
mk
mj Iq λ
k
yj
Pixels
Measured Data
jqkλ
![Page 40: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/40.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
40
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
Transmission TomographyTransmission Tomography
![Page 41: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/41.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
41
Comments on DetailsComments on Details
•• Information geometry basisInformation geometry basis
•• Easily extended to arbitrary Easily extended to arbitrary HH
•• Low density Low density sparsesparse
•• Constraints in iterative decoding vs. forward Constraints in iterative decoding vs. forward modelmodel
•• Performance is intimately connected to Performance is intimately connected to graphgraph
![Page 42: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/42.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
42
OutlineOutline• DSP ISP
- Data models, computational models, algorithms- Central role of information
• Graphical Data Models- X-ray CT imaging- Iterative decoding
• Message Passing EM Algorithms
• Applications Revisited- Iterative decoding- X-ray CT imaging
• Speculation on Trends• Conclusions
![Page 43: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/43.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
43
Science and technologyScience and technologypotentially yield potentially yield
6 Tb/in6 Tb/in22
courtesy R. S. courtesy R. S. IndeckIndeck
Patterned Magnetic Media
![Page 44: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/44.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
44
Advanced Recording MediaAdvanced Recording MediaBluBlu--Ray DiscRay Disc
Next-generation Optical Disc Video Recording Format
![Page 45: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/45.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
45
2D Intersymbol Interference2D Intersymbol Interference
111111
11111111
21
21
11211
ΛΛΛ
ΜΟΜΜΟ
ΛΛΛ
kkkk
k
xxx
xxxx
1111110
110
12
1111110
100100
+++++
+
+
+
+
kkkkkk
kkkkkk
k
kk
k
rrrrrrrr
rrrrrrrr
ΛΛΛΛ
ΜΟΜΜΟΜΜ
ΛΛΛΛΛw(i,j)
⎟⎟⎠
⎞⎜⎜⎝
⎛=
25.05.05.01
h ⊕
Includes Guard Band
jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0 ++++= −−−−
Singla et al., “Iterative decoding and equalization for 2-D recording channels,” IEEE Trans. Magn., Sept. 2002.
![Page 46: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/46.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
46
Full Graph Message PassingFull Graph Message Passing
Measured Data Nodes (r)
Codeword Bit Nodes (x)
Check Nodes (z)
jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0 ++++= −−−−⎟⎟⎠
⎞⎜⎜⎝
⎛=
25.05.05.01
h
![Page 47: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/47.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
47
Full Graph Message PassingFull Graph Message Passing
•• Codeword bit nodes to check nodesCodeword bit nodes to check nodes
•• Check nodes to codeword bit nodes Check nodes to codeword bit nodes
∑∑∈
−→
∈
−→→ +=
zxNz
lxz
xNm
lxm
lzx LLL
\)('
)1('
)(
)1()(
∏∈
−→→ −=
xzNx
lzxz
lxz LL
\)('
)1('
)(
2tanh)1(
2tanh
![Page 48: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/48.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
48
•• Codeword bit nodes to measured data nodesCodeword bit nodes to measured data nodes
•• Measured data nodes to codeword bit nodesMeasured data nodes to codeword bit nodes
∑∑∈
→∈
−→→ +=
)(
)(
\)('
)1('
)(
xNz
lxz
mxNm
lxm
lmx LLL
})\)(':({ )('
)( xmNxLfL lmx
lxm ∈= →→
Full Graph Message PassingFull Graph Message Passing
![Page 49: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/49.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
49
Full Graph Message Passing
-6
-5
-4
-3
-2
-1
0
0 0.5 1 1.5 2 2.5 3
SNR [dB]
Bit
erro
r rat
e, L
og B
ase
10
ISI-freeFull Graph_50
Full Graph Message PassingFull Graph Message Passing[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix
![Page 50: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/50.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
50
Full Graph AnalysisFull Graph Analysis
Length 4 cycles present which degrade performance Length 4 cycles present which degrade performance of messageof message--passing algorithm passing algorithm
x(i+2,j) x(i+2,j+1)
x(i+1,j) x(i+1,j+1)
x(i,j) x(i,j+1)
x(i+2,j+2)
x(i+1,j+2)
x(i,j+2)
r(i+1,j+1)
r(i,j+1)r(i,j)
r(i+1,j)
FromCheckNodes
Kschischang et al., “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform. Theory, Feb. 2001.
![Page 51: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/51.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
51
Ordered Subsets Message Passing Ordered Subsets Message Passing From Imaging From Imaging –– Data set is grouped into subsets to Data set is grouped into subsets to increase rate of convergence of image increase rate of convergence of image reconstruction algorithmsreconstruction algorithms
For Decoding For Decoding –– Measured data is grouped into Measured data is grouped into subsets to eliminate short length cycles in the subsets to eliminate short length cycles in the Channel ISI graphChannel ISI graph
H. M. Hudson, and R. S. Larkin, “Accelerated image reconstruction using ordered subsets of projection data,” IEEE Trans. Medical Imaging, Dec. 1994
![Page 52: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/52.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
52
Labeling of data nodes into 4 subsetsLabeling of data nodes into 4 subsetsFor each iteration use data nodes of one label onlyFor each iteration use data nodes of one label only
Labeled ISI GraphLabeled ISI Graph
J. A. O’Sullivan, and N. Singla, “Ordered subsets message-passing,” Int’l Symp. Inform. Theory, Yokohama, Japan 2003.
![Page 53: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/53.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
53
Ordered Subsets Message Passing
-6
-5
-4
-3
-2
-1
0
0 0.5 1 1.5 2 2.5 3SNR [dB]
Bit
erro
r rat
e in
log1
0
ISI-freeOrdered Subsets_200Full Graph_50
[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrixOrdered Subsets Message PassingOrdered Subsets Message Passing
![Page 54: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/54.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
54
CT Imaging in Presence of High CT Imaging in Presence of High Density Attenuators (J. Williamson, PI)Density Attenuators (J. Williamson, PI)
Brachytherapy applicators After-loading colpostats
for radiation oncology
Cervical cancer: 50% survival rateDose prediction important
Object-Constrained Computed Tomography (OCCT)
![Page 55: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/55.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
55
Filtered Back ProjectionFiltered Back Projection
Truth FBP
FBP: inverse Radon transform
![Page 56: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/56.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
56
Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x• Data d(y) Poisson, means g(y:µ), log likelihood function
• Mean unattenuated counts I0, mean background β• Attenuation function µ(x,E), E energies
• Maximize over µ or ci; equivalently minimize I-divergence
)(),(),(exp),():(
):():(ln)()):(|(
0 yExxyhEyIyg
ygygydgdl
E x
y
βµµ
µµµ
+⎟⎟⎠
⎞⎜⎜⎝
⎛−=
−=⋅
∑ ∑
∑
∈
∈
X
Y
∑=
=I
iii ExcEx
1)()(),( µµ
![Page 57: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/57.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
57
MaximumMaximum--Likelihood Likelihood Minimum IMinimum I--divergencedivergence
• Poisson distribution
• Poisson distributed data loglikelihood function
• Maximization over µ equivalent to minimization of I-divergence
λλ
λ
λλ
λ λ
+−=
−−==
== −
kkkkI
kkkNP
ek
kNPk
ln)||(
!lnln)(ln!
)(
)(),(),(exp),():(
):()():(
)(ln)()):(||(
):():(ln)()):(|(
0 yExxyhEyIyg
ygydyg
ydydgdI
ygygydgdl
E x
y
y
βµµ
µµ
µ
µµµ
+⎟⎠
⎞⎜⎝
⎛−=
+−=⋅
−=⋅
∑ ∑
∑
∑
∈
∈
∈
X
Y
Y
![Page 58: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/58.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
58
Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence
Difficulties: log of sum, sums in exponent
)()()(),(exp),():(
):()():(
)(ln)()):(||(
):():(ln)()):(|(
10 yExcxyhEyIyg
ygydyg
ydydgdI
ygygydgdl
E x
I
iii
y
y
βµµ
µµ
µ
µµµ
+⎟⎟⎠
⎞⎜⎜⎝
⎛−=
+−=⋅
−=⋅
∑ ∑ ∑
∑
∑
∈ =
∈
∈
X
Y
Y
![Page 59: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/59.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
59
Interpretation: Compare predicted data to measured data via ratio of backprojectionsUpdate estimate using a normalization constant
Comments: choice for constants; monotonic convergence;linear convergence; fixed points satisfy Kuhn-Tucker conditions; constraints easily incorporated
)(~)(ˆ
ln)(
1)(ˆ)(ˆ)(
)()()1(
xbxb
xZxcxc l
i
li
i
li
li −=+
New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography
![Page 60: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/60.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
60David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000001
![Page 61: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/61.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
61David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000002
![Page 62: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/62.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
62David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000005
![Page 63: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/63.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
63David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000010
![Page 64: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/64.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
64David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000020
![Page 65: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/65.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
65David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000050
![Page 66: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/66.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
66David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000100
![Page 67: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/67.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
67David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000200
![Page 68: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/68.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
68David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0000500
![Page 69: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/69.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
69David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0001000
![Page 70: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/70.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
70David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0002000
![Page 71: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/71.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
71David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0005000
![Page 72: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/72.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
72David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0010000
![Page 73: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/73.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
73David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0020000
![Page 74: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/74.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
74David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0050000
![Page 75: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/75.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
75David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0100000
![Page 76: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/76.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
76David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0200000
![Page 77: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/77.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
77David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 0500000
![Page 78: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/78.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
78David G. PolitteOctober 31, 2002
Mini CT, AM Iteration 1000000
![Page 79: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/79.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
79
Our Plans in CT ImagingOur Plans in CT Imaging
•• CT CT MultirowMultirow SinogramSinogram data: data: 1408 1408 ×× 768 768 ×× nndd ×× nnzz
– where nd is the number of detector rows and nz is the number of gantry rotations
•• Fully 3Fully 3--D Implementations for D Implementations for Quantitative CTQuantitative CT
•• SpeedSpeed--up: Ordered Subsets, up: Ordered Subsets, MultigridMultigrid Methods, Parallel Methods, Parallel Implementations on Clusters Implementations on Clusters of PCsof PCs
•• Future: PETFuture: PET--CT CT Siemens Somotom Emotion
![Page 80: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/80.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
80
Slide and data from R. Laforest and M. Mintun.
PETCT-211
![Page 81: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/81.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
81
PETCT-165
Slide and data from R. Laforest and M. Mintun.
![Page 82: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/82.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
82
Additional Algorithm/DetectorAdditional Algorithm/DetectorModel DevelopmentModel Development
•• RegularizationRegularization•• Energy integrating detectors Energy integrating detectors •• Finite detector size, better source modelFinite detector size, better source model•• Finite pixel, Finite pixel, voxelvoxel sizesize•• Average integral or average exponentialAverage integral or average exponential
(arithmetic vs. geometric average)(arithmetic vs. geometric average)•• Partial volume effectsPartial volume effects•• MotionMotion•• ScatteringScattering•• Limited angle tomographyLimited angle tomography•• Region of interestRegion of interest•• Scanner implementations: beam hardening Scanner implementations: beam hardening
correction, sampling, etc.correction, sampling, etc.
∫ ),( EyEdN
![Page 83: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/83.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
83
Computation andCommunication
Signal Processing
Information Theory
ComplexityComplexity--constrained processingconstrained processingSignal processing on graphsSignal processing on graphsDistributed signal processingDistributed signal processingDistributed information theoryDistributed information theoryDistributed computation and Distributed computation and
communicationcommunicationOptimal information extraction, Optimal information extraction,
communication, computationcommunication, computation
![Page 84: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/84.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
84
Limits of Information TheoryLimits of Information Theory•• Information theory provides bounds on performance Information theory provides bounds on performance
of communication, compression, and data analysisof communication, compression, and data analysis– Channel coding theorem (capacity)– Entropy, rate-distortion theory– Fisher information
•• Open Problems in Information TheoryOpen Problems in Information Theory– Broadcast channel p(y1,y2|x) capacity region of achievable (R1,R2)
» Depends only on p(y1|x) and p(y2|x); degraded channel known– Distributed source compression achievable (R1,R2, D1,D2)
•• Algorithmic information theory (Algorithmic information theory (KolmogorovKolmogorovcomplexity)complexity)
p(x,yp(x,y))XXnn
YYnn
XXnn
ffyy
ffxxYYnngg ^
^
![Page 85: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/85.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
85
Speculation on FutureSpeculation on Future•• Distributed compressionDistributed compression
– Compression with side information– Analogies to information embedding– Reduced communication rates
•• Broadcast channel models Broadcast channel models – Appropriate for motes– Communication at different rates using a common signal– Reduced communication rates
•• Tradeoffs in communication and computationTradeoffs in communication and computation
•• Mobile computing: cheap Mobile computing: cheap expensive expensive cheapcheap– mobile base station + network mobile
![Page 86: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/86.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
86
ConclusionsConclusions
•• Information Signal ProcessingInformation Signal Processing—— DSP, Information Theory, Computation and DSP, Information Theory, Computation and
CommunicationCommunication
•• Role of Graphical ModelsRole of Graphical Models•• Message Passing EM AlgorithmsMessage Passing EM Algorithms•• Iterative Equalization and DecodingIterative Equalization and Decoding•• XX--Ray CT ImagingRay CT Imaging
![Page 87: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/87.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
87
![Page 88: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/88.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
88
Future WorkFuture Work
Signal Processing
Information Theory
Fast algorithmsFast algorithms
Optimal communicationOptimal communication
Distributed information Distributed information theory theory
Computation andCommunication
![Page 89: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/89.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
89
MultigridMultigrid and and MultiresolutionMultiresolutionComment: NIBIB ProposalComment: NIBIB Proposal
•• MotivationMotivation– Speed of computations– Multiresolution capabilities– Regularization with adaptive resolution– Complexity regularization
•• C. C. BoumanBouman, Oh, et al., Purdue, Oh, et al., Purdue•• Surrogate function viewSurrogate function view
– Original function difficult to minimize directly– Upper bound using a convex function– Minimize upper bound
•• New ideasNew ideas– Guaranteed monotonicity in multigrid– Multigrid alternating minimization algorithms
•• DifficultyDifficulty– Dual representations in parameters and means
![Page 90: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/90.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
90
MultigridMultigrid BasisBasis
)()()(),(exp),():(
):()():(
)(ln)()):(||(
10 yExcxyhEyIyg
ygydyg
ydydgdI
E x
I
iii
y
βµµ
µµ
µ
+⎟⎠
⎞⎜⎝
⎛−=
+−=⋅
∑ ∑ ∑
∑
∈ =
∈
X
Y
[ ] ( )sother term
)()(exp)(
1)(ˆ)()()(~)||(
)()()(
)(),(exp)()(),(exp),(
)()()()(),(exp),(),(
),(),(),(),(ln),()||(minmin
)1(
1
)1()()(
1
)1(
1
)(0
10
+
∆−+∆+≤
⎟⎟⎠
⎞⎜⎜⎝
⎛∆−⎟
⎠
⎞⎜⎝
⎛−
=⎟⎟⎠
⎞⎜⎜⎝
⎛−=
+−=
+
∈ =
+
∈ =
+
∈ =
∈ =
∈∈
∑∑
∑∑∑ ∑
∑ ∑
∑∑
xcxZxZ
xbxcxcxbqpI
xcxZxZ
ExyhExcxyhEyI
ExZxZxcxyhEyIEyq
EyqEypEyqEypEypqpI
kii
x
I
i ii
ki
kii
k
x
I
i
kii
i
i
x
I
ii
ki
x
I
ii
i
ii
y Epq
X
XX
X
YL
µµ
µ
Restrict updates to a subsetAll inequalities hold
![Page 91: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/91.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
91
DSP DSP ISPISP
•• SamplingSampling– Uniform shifts xn = x(t-nT)– Filtering: Fourier transform (FFT) – linearity, implied stationarity– Time space space-time
•• DistributedDistributed– Signals– Sensing– Computation
∑−
=
−=1
0
/2N
n
Nknjnk exX π
∑−
=
−=1
0
||,2N
n
Nnkjnk exX π
X(0)X(1)X(2)X(3)X(4)X(5)X(6)X(7)
x(0)x(4)x(2)x(6)x(1)x(5)x(3)x(7)
![Page 92: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/92.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
92
![Page 93: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/93.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
93
![Page 94: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/94.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
94
![Page 95: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/95.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
95
![Page 96: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/96.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
96
![Page 97: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/97.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
97
DSP DSP ISPISP
•• Digital Signal ProcessingDigital Signal Processing
•• DiscreteDiscrete--Time Signal ProcessingTime Signal Processing
•• ‘‘DistributedDistributed’’ Signal ProcessingSignal Processing
•• Information Theory Information Theory Bits, BitBits, Bit--Rates, DistortionRates, Distortion
•• ‘‘DistributedDistributed’’ Information TheoryInformation Theory
•• Information Signal ProcessingInformation Signal Processing
![Page 98: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/98.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
98
Signal Processing
Information Theory Computations
![Page 99: Information Signal Processingjao/Talks/InvitedTalks/UMNTalk052504.pdf · Information Signal Processing J. A. O’Sullivan. 05/25/2004 9 Computation and Communication Information Signal](https://reader033.vdocuments.us/reader033/viewer/2022041711/5e4851ae4af74825fd5c42f7/html5/thumbnails/99.jpg)
J. A. O’Sullivan. 05/25/2004Information Signal Processing
99
Signal Processing
Information Theory Computations