a transform-based variational framework guy gilboa pixel club, november, 2013
TRANSCRIPT
![Page 1: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/1.jpg)
A Transform-based Variational Framework
Guy Gilboa
Pixel Club, November, 2013
![Page 2: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/2.jpg)
In a Nutshell
Spatial Input
Transform Analysis
Transform Filtering
Spatial Output
𝐼Φ→𝑆𝐻
→𝑆𝐻Φ
−1
→𝐼
Fourier inspiration:
Fourier Scale Fourier Scale
𝐹→
𝐿𝑃𝐹→
𝐹−1
→
Spectral
TV Flow
0 20 40 60 80 100 120 1400
500
1000
1500
2000
2500
3000
TV ScaleTV Scale
0 20 40 60 80 100 120 1400
500
1000
1500
2000
2500
3000
![Page 3: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/3.jpg)
Relations to eigenvalue problemsGeneral linear: (L linear operator)Functional based
uLu
)div( uuu || 21 dxuJH
|| dxuJTV ||
div uu
u
Linear
Nonlinear
![Page 4: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/4.jpg)
What can a transform-based approach give us?Scale analysis based on the
spectrum.New types of filtering – otherwise
hard to design: nonlinear LPF, BPF, HPF.
Nonlinear spectral theory – relation to eigenfunctions and eigenvalues.
Deeper understanding of the regularization, optimal design with respect to data, noise and artifacts.
![Page 5: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/5.jpg)
Examples of spectral applications today:Eigenfunctions for 3D processing
Taken from Zhang et al, “Spectral mesh processing”, 2010.
Taken from L Cai, F Da, “Nonrigid deformation recovery..”, 2012.
![Page 6: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/6.jpg)
Image Segmentatoin
Eigenvectors of the graph Laplacian[Taken from I. Tziakos et al, “Color image segmentation using Laplacian eigenmaps”, 2009 ]
![Page 7: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/7.jpg)
Some Related StudiesAndreu, Caselles, Belletini, Novaga et al 2001-
2012– TV flow theory.Steidl et al 2004 – Wavelet – TV relationBrox-Weickert 2006 – scale through TV-flowLuo-Aujol-Gousseau 2009 – local scale measuresBenning-Burger 2012 – ground states (nonlinear
spectral theory)Szlam-Bresson – Cheeger cuts.Meyer, Vese, Osher, Aujol, Chambolle, G.
and many more – structure-texture decomposition.Chambolle-Pock 2011, Goldstein-Osher 2009 –
numerics.
![Page 8: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/8.jpg)
Scale Space – a Natural Way to Define Scale
We’ll talk specifically about total-variation (TV-flow, Andreu et al - 2001):
)( ,| , 0 uJpfupu utt
xxfxun
u
Du
Du
t
u
in ),();0(
),0(on ,0
),0(in ,||
div
Scale space as a gradient descent:
![Page 9: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/9.jpg)
TV-Flow:A behavior of a disk in time[Andreu-Caselles et al–2001,2002, Bellettini-Caselles-Novaga-2002, Meyer-2001]
Center of disk, first and second time derivatives:
t
… …
𝑢 𝑢𝑡 𝑢𝑡𝑡2
0 −0 .5
0
0
![Page 10: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/10.jpg)
Spectral TV basic framework
Phi(t) definition
txtxt utt );();(
![Page 11: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/11.jpg)
Reconstruction
Reconstruction formula
Th. 1: The reconstruction formula recovers
fdttf
0
)(ˆ
dxxff )(||
1
![Page 12: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/12.jpg)
Spectral response
Spectrum S(t) as a function of time t:
dxxtL
xttS |);(|);()( 1
t
0 20 40 60 80 100 120 1400
500
1000
1500
2000
2500
3000
S(t)f
![Page 13: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/13.jpg)
Spectrum example
0 5 10 15 20 25 30 35 40 45 500
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2x 10
4
tS
(t)
f S(t)
![Page 14: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/14.jpg)
Dominant scales
0 5 10 15 20 25 30 35 40 45 500
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2x 10
4
t
S(t
)
);2( xt );10( xt );37( xt
![Page 15: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/15.jpg)
Eigenvalue problem
The nonlinear eigenvalue problem with respect to a functional J(u) is defined by:
We’ll show a connection to the spectral components .
),(
,
uJp
up
)(t
![Page 16: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/16.jpg)
Solution of eigenfunctions
Th. 2: For is an eigenfunction with eigenvalue then:
1)()
1()(
)()1
();(
1
,0
10 ),1)((
);(
LxfttS
xftxt
t
ttxfxtu
![Page 17: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/17.jpg)
What are the TV eigenfunctions?In 2D, is a characteristic function of a convex set. I then is an eigenfunction.
Area
Perimeterboundary)on curvaturemax(
𝜅 (𝑝 )=1𝑟;𝑃 (𝐶 )|𝐶|
=2𝜋𝑟𝜋𝑟2
=2𝑟 [Giusti-1978], [Finn-1979],[Alter-
Caselles-Chambolle-2003].
![Page 18: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/18.jpg)
Filtering
Let H(t) be a real-valued function of t. The filtered spectral response is
)();(:);( tHxtxtH
fdtxtxf HH
0
);()(
The filtered spatial response is
𝜙(𝑡) 𝜙𝐻 (𝑡)H(t)
![Page 19: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/19.jpg)
Filtering, example 1:
TV Band-Pass and Band-Stop filters
Band-pass Band-stop0 2 4 6 8 10 12 14
0
0.5
1
1.5
2
2.5x 10
4
f S(t)
![Page 20: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/20.jpg)
Disk band-pass example
0 20 40 60 80 100 120 1400
500
1000
1500
2000
2500
3000
S(t)
![Page 21: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/21.jpg)
We have the basic framework
Spatial Input
Transform Analysis
Transform Filtering
Spatial Output
𝐼Φ→𝑆𝐻
→𝑆𝐻Φ
−1
→𝐼
txtxt utt );();(
LxttS 1);()(
0 20 40 60 80 100 120 1400
500
1000
1500
2000
2500
3000
fdtxtxf HH
0
);()(
![Page 22: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/22.jpg)
Numerics Many ways to solve.Variational approach was chosen:
Currently use Chambolle’s projection algorithm (some spikes using Split-Bregman, under investigation).
In time: ◦ 2nd derivative - central difference◦ 1st derivative - forward differnce◦ Discrete reconstruction algorithm proved for
any regularizing scale-space (Th. 4).
0))1(()()1( nutpnunu ||)(||2
1)(
2
2nuut
uJL
)(uput
![Page 23: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/23.jpg)
TV-Flow as a LPFTh. 3: The solution of the TV-flow is equivalent to spectral filtering with:
0 10 20 30 40 50 60 70 80 90 1000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
t
HT
FV
,t1
t1 = 1t1 = 5t1 = 10t1 = 20
,
0 ,0
11
1
, 1
ttt
tt
ttH tTVF
![Page 24: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/24.jpg)
Nonlocal TVReminder: NL-TV (G.-Osher
2008):
Gradient
Functional
),()()()( yxwxuyuxuw
dxxuuJ wTVNL |)(|)(
yx,
![Page 25: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/25.jpg)
Spectral NL-TV?The framework can fit in principle
many scale-spaces, like NL-TV flow. We can obtain a one-homogeneous regularizer.
What is a generalized nonlocal disk?What are possible eigenfunctions? It is expected to be able to process
better repetitive textures and structures.
![Page 26: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/26.jpg)
Sparseness in the TV senseSparse spectrum – the signal
has only a few dominant scales.
Or many small ones (here TV energy is large)
Can be a large objects
Natural images – are not very sparse in general
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
500
1000
1500
2000
2500
S(t)
t
0 5 10 15 20 25 30 35 40 45 500
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2x 10
4
t
S(t
)
0 20 40 60 80 100 120 1400
500
1000
1500
2000
2500
3000
![Page 27: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/27.jpg)
Noise Spectrum
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
200
400
600
800
1000
1200
1400
S(t)
t
𝑆 (𝑡)→
Various standard deviations:
S(t)
![Page 28: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/28.jpg)
Noise + signal
Not additive. Spreads original image spectrum. Needs to be investigated.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
500
1000
1500
2000
2500
Clean image
Noise only
Image with noiseuf
f-u
Band-pass filtered
0 5 10 15 20 250
2000
4000
6000
8000
10000
12000
t
Signal
Noise
Signal + Noise
![Page 29: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/29.jpg)
Spectral Beltrami Flow?Initial trials on Beltrami flow with parameterization such that it is closer to TVOriginal Beltrami Flow Spectral
Beltrami
Difference images:
• Keeps sharp contrast
• Breaks extremum principle
Values along one line (Green channel)
0 20 40 60 80 100 1200.4
0.5
0.6
0.7
0.8
0.9
1
1.1
Original
Spectral Beltrami
Beltrami Flow
Spectral Beltrami
![Page 30: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/30.jpg)
Segmentation priorsSwoboda-Schnorr 2013 –
convex segmentation with histogram priors.
We can have 2D spectrum with histograms
Use it to improve segmentation
S(t,h)
![Page 31: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/31.jpg)
Texture processingMany texture bands
We can filter and manipulate certain bands and reconstruct a new image.
Generalization of structure-texture decomposition.
t
tdtt
i
i
1
)(Band(i) 1
,..2,1
ii tt
i
![Page 32: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/32.jpg)
Processing approachDeconstruct the image into bands
Identify salient textures
Amplify / attenuate / spatial process the bands.
Reconstruct image with processed bands
![Page 33: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/33.jpg)
Color formulation
Vectorial TV – all definitions can be generalized in a straightforward manner to vector-valued images.
Bresson-Chan (2008) definition and projection algorithm is used for the numerics.
![Page 34: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/34.jpg)
Orange example
![Page 35: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/35.jpg)
Orange – close up
Original Modes 2,3=0 Modes 2-5=x1.5
![Page 36: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/36.jpg)
Selected phi(t) modes (1, 5, 15, 40)
residual
f
![Page 37: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/37.jpg)
Old man
![Page 38: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/38.jpg)
Old man – close up
Original 2 modes attenuated 7 modes attenuated
![Page 39: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/39.jpg)
Old Man - First 3 Modes
Modes: 1 2 3
![Page 40: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/40.jpg)
Take Home Messages Introduction of a new TV
transform and TV spectrum. Alternative way to understand
and visualize scales in the image.
Highly selective scale separation, good for processing textures.
Can be generalized to other functionals.
![Page 41: A Transform-based Variational Framework Guy Gilboa Pixel Club, November, 2013](https://reader035.vdocuments.us/reader035/viewer/2022081514/55199b285503463d068b49bc/html5/thumbnails/41.jpg)
Thanks!
Refs. Google “Guy Gilboa publications”• Preliminary ideas are in SSVM 2013 paper. • Most material is in CCIT Tech report 803.• Up-to-date and organized - submitted journal
version – contact me.