statistics of natural images
DESCRIPTION
Statistics of natural images. May 30, 2010 Ofer Bartal Alon Faktor. Outline. Motivation Classical statistical models New MRF model approach Learning the models Applications and results. Motivation. Big variance in appearance Can we even dream of modeling this?. Motivation. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/1.jpg)
1
Statistics of natural images
May 30, 2010Ofer BartalAlon Faktor
![Page 2: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/2.jpg)
2
Outline
• Motivation• Classical statistical models• New MRF model approach• Learning the models• Applications and results
![Page 3: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/3.jpg)
3
Motivation
• Big variance in appearance • Can we even dream of modeling this?
![Page 4: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/4.jpg)
4
Motivation
• Main questions:– Do all natural images obey some common
“rules”?– How can one find these “rules”?– How to use “rules” for computer vision
tasks?
![Page 5: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/5.jpg)
5
Motivation
• Why bother to model at all?
• “Noise”, uncertainty
• Model helps choose the “best” possible answer
• Lets see some examples
Natural image model
![Page 6: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/6.jpg)
6
Noise-blur removal
• Consider the classical De-convolution problem
• Can be formulated as linear set of equations:
?Y h X N X
cs cs csy Hx n
H Y+X
N
![Page 7: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/7.jpg)
7
X
=
YNh
X̂
?
Noise-blur removal
![Page 8: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/8.jpg)
8
Inpainting
Y AX n
Y
?X
1 0 0 ... 00 1 0 0 ... 00 0 0 0 0 1 0 0 ... 00 0 0 0 0 0 1 0 ... 00 0 .... 0 1
A
Missing lines of identity matrix = missing pixels (under-determined system)
![Page 9: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/9.jpg)
9
Motivation
• Problems: – Unknown noise– H may be singular (Deconvolution)– H may be under-determined (Inpainting)
• So there can be many solutions. • How can we find the “right” one?
![Page 10: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/10.jpg)
10
Motivation
• Goal: Estimate x– Assume:
• Prior model of natural image:• Prior model of noise:
– Use MAP estimator to find x:
* arg max ( | ) arg max ( | ) ( )x x
x P x y P y x P x
( )xP x
* arg max ( ) ( )n xx
x P y Hx P x
( )nP n
![Page 11: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/11.jpg)
11
Energy Minimization problem
• The MAP problem can be reformulated as:
data term( | )+prior term( ) ˆ arg min
x
E y x xx E
E
x
![Page 12: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/12.jpg)
13
Classical models
• Smoothness prior (model of image gradients) – Gaussian prior (LS problem)– L1 Prior and sparse prior (IRLS problem)
Image gradient
![Page 13: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/13.jpg)
14
Gaussian Priors
• Assume:
– Gaussian priors on gradients of x:
– Gaussian noise:• Using this assumption:
2
221( )2
x
p x e
* arg min 2T T
xx x Tx x b
* arg max ( ) ( )n xx
x P y Hx P x
2~ (0, )n N
![Page 14: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/14.jpg)
15
Non-Gaussian Priors
• Empirical results: image gradients have a Non-Gaussian heavy tailed distribution
• We assume L1 or sparse prior• We solve it by IRLS –iterative re-weighted LS
![Page 15: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/15.jpg)
16
De-convolution Results
Gaussian prior Sparse priorBlurred image
Good results on simple images
![Page 16: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/16.jpg)
17
De-noising Results
De-noising resultNoisy image
Poor results on real natural images
![Page 17: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/17.jpg)
18
Classical models – Pro’s and Con’s
• Advantages:– Simple and easy to implement
• Disadvantages:– Too Heuristic– Only one property - Smoothness– Bias towards totally smooth images:
P P
![Page 18: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/18.jpg)
19
Going Beyond Classical Models
0 1 2 3 40
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
number of similar patches (in log10 scale)
prob
abilt
y
0 1 2 3 40
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
number of similar patches (in log10 scale)
prob
abilt
y
0 1 2 3 40
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
number of similar patches (in log10 scale)
prob
abilt
y
![Page 19: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/19.jpg)
20
Modern Approach
• Model is based on image properties• Choose properties using image dataset
• Questions:1. What types of properties?
Responses to linear filters.2. How to find good properties?
Either pre-determined bank or learn from data.3. How should combine properties to one distribution?
We will see how.
![Page 20: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/20.jpg)
21
Mathematical framework
• Want: A model p(I) of real distribution f(I).• Computationally hard:
– A 100x100 pixel image has 10,000 variables• Can explicitly model only a few dimensions at a time
Arrow = viewpoint of few dimensions
![Page 21: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/21.jpg)
22
Mathematical framework
• A viewpoint is a response to a linear filter• A distribution over these responses is a
marginal of real distribution f(I)• (Marginal = Distribution over a subset of variables)
Arrow = marginal of f(I)
![Page 22: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/22.jpg)
23
Mathematical framework
• If p(I) and f(I) have the same marginal distributions of linear filters then p(I)=f(I) (proposition by Zhu and Mumford)
• “Hope”: If we will choose K “good” filters then p(I) and f(I) will be “close”.
How do we measure “close?”
![Page 23: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/23.jpg)
24
Distance between distributions
• Kullback-Leibler divergence:
• Problem - f(I) unknown• Proposition - use instead:
• Measures fit of model to observations
( ), ( ; , ) log ( ) log ( ; , )f fKL f I p I S E f I E p I S
~
( ), ( ; , ) log ( ; , ) log ( ; , )P X
KL f I p I S p I S p I S
( ; , )p I S X
![Page 24: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/24.jpg)
25
Illustration
log ( ; , )p I S
log ( ; , )p I S
~KL
~
( ), ( ; , ) log ( ; , ) log ( ; , )P X
KL f I p I S p I S p I S
![Page 25: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/25.jpg)
26
Getting synthesized images
• Get synthesized images by sampling the learned model
• Sample using Markov Chain Monte Carlo (MCMC).
• Drawback: Learning process is slow
xp
![Page 26: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/26.jpg)
27
Our model P(I) – A MRF
• MRF = Markov Random Field• A MRF is based on a graph G=(V,E).
V – pixels E – between pixels that affect each other
• Our distribution is the MRF:
( )1( ) exp ( )c c
c Cliques
p I U IZ
![Page 27: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/27.jpg)
28
Simple grid MRF
• Here, cliques are edges• Every pixel belongs to 4 cliques
![Page 28: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/28.jpg)
29
MRF
• We limit ourselves to:
– Cliques of fixed size (over-lapping patches)
– Same for all cliques
• We get:
( ) ( )( ) ( )
1
( ) ( )K
Tc cU I F I
( ) ( )( )
1 1
1( ) exp ( )C K
Tc
c
p I F IZ
U
![Page 29: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/29.jpg)
30
MRF simulation
( ) ( )( )
1 1
1( ) exp ( )C K
Tc
c
p I F IZ
![Page 30: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/30.jpg)
31
Histogram simulation
( ) ( )obsnH I
Histogram of a marginal
![Page 31: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/31.jpg)
32
MRF
• In terms of convolutions:
• Denote: Set of potential functions:
• Denote: Set of filters:
( )
1...K
( )
1...KS F
( ) ( )
1 ( , )
,1( ; , )
K
x y
F I x y
p I S eZ
![Page 32: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/32.jpg)
33
MRF - A simple example
• Cliques of size 1• Pixels are i.i.d and distributed by grayscale histogram
grayscale histogram
Drawback: cliques are too small
![Page 33: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/33.jpg)
34
MRF - Another simple example
• Clique = whole image• Result: Uniform distribution on images in dataset
Px
Drawback: cliques are too big
![Page 34: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/34.jpg)
37
Revisiting classical models
• Actually, the classical model is a pairwise MRF:
• Has cliques of size 2:
• Has only 2 linear filters => 2 marginals
• No guarantee that p(I) will be close to f(I)
( , )( ( , )) ( ( , ))1( ) x yx y
I x y I x yp I e
Z
![Page 35: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/35.jpg)
39
Zhu and Mumford’s approach (1997)
• We want to find K “good” filters• Strategy:
– Start off with a bank B of possible filters– Choose subset that minimizes the
distance between p(I) and f(I)– For computational reasons, choose filters one by
one using a greedy method
, | |S B S K
![Page 36: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/36.jpg)
41
Choosing the next filter
• AIG = the difference between the model p(I) and the data from the viewpoint of marginal
• AIF = the difference in between different images in dataset from the viewpoint of marginal
( ) ( ) ( )IC AIG AIF
( ) ( )( ; , )
1
( ) ( )
1
1( ) ( ) ( )21( ) ( )
2obs
Mobsn P I S
n
Mobsn
n
AIG H I E H IM
AIF H IM
![Page 37: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/37.jpg)
42
Algorithm – Filter selection
Bank of filters
IC
IC
IC arg max
max
Model ( )learn
![Page 38: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/38.jpg)
44
Learning the potentials
( ) Model
( )IC
Calculate update
Init
(Using maximum entropy on P)
![Page 39: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/39.jpg)
45
The bank of filters
• Filter types: – Intensity filter (1X1)– Isotropic filters - Laplacian of Gaussian (LG, )– Directional filters - Gabor (Gcos, Gsin)
• Computation in different scales - image pyramid
Laplacian of Gaussian Gabor
![Page 40: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/40.jpg)
46
Running example of algorithmExperiment I
Use only small filters
![Page 41: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/41.jpg)
47
Results
All learned potentials have a diffusive nature
( ) ( )( )
1 1
1( ) exp ( )C K
Tc
c
p I F IZ
![Page 42: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/42.jpg)
48
Running example of algorithmExperiment II
• Only gradient filters, in different scales• Small filters -> diffusive potential (as expected)• Surprisingly: Large filters -> reactive potentials
Diffusive Reactive
![Page 43: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/43.jpg)
50
Examples of the synthesized images
Experiment I Experiment II
This image is more “natural” because it has some regions with sharp boundaries
![Page 44: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/44.jpg)
51
Outline
• We have seen:– MRF models – Selection of filters from a bank – Learning potentials
• Now:– Data-driven filters – Analytic results for simple potentials– Making sense in results– Applications
![Page 45: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/45.jpg)
52
Roth and Black’s approach
filters potentials
Chosen from bank Learn a-parametricallyX XLearn from data Learn parametrically
Learn together
![Page 46: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/46.jpg)
53
Motivation – model of natural patches
• Why learn filters from data?• Inspiration from models of natural patches:
– Sparse coding– Component analysis– Product of experts
![Page 47: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/47.jpg)
54
Motivation – Sparse Coding of patches
• Goal: find a set s.t.
•
• Learn from database of natural patches
• Only few filters should fire on a given patch
1
, are sparseN
i i ii
patch a F a
iF
,i ia patch F
iF
1 2 3 4 5
![Page 48: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/48.jpg)
55
Motivation – Component analysis
• Learn by component analysis:– PCA– ICA
• Results in “filters like” components– PCA – first components look like contrast filters– ICA - components look like Gabor filters
iF
![Page 49: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/49.jpg)
56
PCA results
high
low
![Page 50: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/50.jpg)
57
ICA results
• Independent filters • Can derive model for patches:
1
( ) ( )n
Ti i
i
P x p F x
TiF x
ip
![Page 51: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/51.jpg)
58
Motivation – Product of experts
• More sophisticated model for natural patches:
• Training of MLE => “intuitive” filters:
2
1
1( ; ) ( ; ), , ( ) 1( 2, )
i
i i i
KT
POE i sti ii
F F zzZ F
p X X
texturecontrast
![Page 52: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/52.jpg)
59
• extension of POE to FOE:
Field of experts (FOE)
( )1 1
1( ; ) exp ( ; ) ,,( , )
C KT
FOE i iii
i ii
cc
F Fp I IFZ
log( )st
( )1 1
( ; )iC K
TFOE i c
c i
E F I
Roth S., Black M. J., Fields of experts IJCV, 2009
![Page 53: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/53.jpg)
60
The experts
• Student-t experts2
( ) 12
i
izz
( )st z
![Page 54: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/54.jpg)
61
Meaning of
• Higher means:– Punishes high responses more severely – A filter with higher weight
( )st z
1
2
( )
1
1( ; , ) exp ( , )
g log 12
K
FOE i i i iii i
TCi c
ic
p I F gZ F
F I
![Page 55: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/55.jpg)
Learning the model
log ( ; , )p I S
log ( ; , )p I S
~KL
Model
1
2K
MCMCinit
random
![Page 56: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/56.jpg)
65
Results of learning FOE
Filters aren’t “intuitive”
F
![Page 57: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/57.jpg)
67
So far…
filters potentials
Chosen from bank Learned a-parametrically
diffusive reactive
Small filters Large filters
non-intuitive?
![Page 58: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/58.jpg)
68
So far…
filters potentials
Learned from database Learned parametrically
non-intuitive?
![Page 59: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/59.jpg)
69
What now?• Revisiting POE and FOE with Gaussian
potentials• Relation to non-Gaussian potentials• Making sense of previous results
Weiss Y., Freeman W. T. What makes a good model of natural images?. CVPR, 2007
![Page 60: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/60.jpg)
70
Gaussian POE
2
2
1
2
1
* 2
1
1; exp ( )( )
ln ; ( ) ln ( )
arg min ( ) ln ( )i
z
KT
GPOE i iii
KT
GPOE i i ii
KT
i i iML F i
e
p x F F xZ F
p x F F x Z F
F F x Z F
![Page 61: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/61.jpg)
71
• Claim: Z is constant for any set of K orthonormal vectors
•
• This has an analytic solution – the K minor components of the data
Gaussian POE
* 2
1
arg min ( )i
KT
i iF orthonormal i
F F x
![Page 62: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/62.jpg)
72
• Non-intuitive high-frequency filters• Reminder - PCA
ResultsExample of learned filters
high
low
![Page 63: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/63.jpg)
73
Gaussian FOE
2
2( )
1 1
2 2( )
1 1 1 ,
2 2 2
1
2
1
1( ;{ }) exp ( )({ })
( ) ( * ) ( , )
{ }( ) { }( ) ( ) { }( )
ln ({ }) ln { }( ) ln ( )
K CT
GFOE i i ci ci
K C KTi c i
i c i x y
K
ii
K
i ii
z
p I F F IZ F
F I F I x y
F I G I
Z F F G
![Page 64: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/64.jpg)
74
Gaussian FOE
* 2
( )1 1
2*
( )
*2
arg min ( ) ln ( )
( ) arg min ( ) { }( ) ln ( )
1( ){ }( )
i
K CT
i i c iML F i c
MLG
ML
F F I Z F
G G I G
GI
![Page 65: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/65.jpg)
75
Gaussian FOE
• satisfies:
=> Optimal filters have high frequencies
2*2
1
1{ }( ){ }( )
K
ii
FI
2*
1
{ }( )K
ii
F
2{ }( )I
*iF
![Page 66: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/66.jpg)
76
• Non-Gaussian potentials -> modeled by GSM
• Properties of GFOE hold for GSM
Gaussian Scale Mixture (GSM)
![Page 67: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/67.jpg)
77
Revisiting FOE
• Student t expert – fit GSM• Filters have the property of
Natural image Roth and Black filters
22
1
1{ }( ){ }( )
K
ii
FI
high-frequency filters
![Page 68: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/68.jpg)
78
Learning FOE with fixed filters
Algorithm prefers high-frequency filters
![Page 69: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/69.jpg)
79
Conclusion
• For Gaussian potentials and GSM’s:learning => High frequency filters
• Experimental evidence to this phenomena • Maybe there is a “logic” behind this non-intuitive
result?
![Page 70: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/70.jpg)
80
Making Sense of results
• Criterion for “good” filters for patches – Rarely fire on natural images and fire frequently on all other images
Patches from Natural images
Histogram of filter responses
White noise
![Page 71: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/71.jpg)
81
Making Sense of results
• An image was modeled by what you don’t expect to find in it
• This is satisfied by the classical prior of smooth gradients
• But why limit ourselves to intuitive filters?• Maybe non-intuitive filters can do better…
![Page 72: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/72.jpg)
82
reactivediffusive
White noise
Patches from Natural images
Revisiting diffusive and reactive potentials
White noise
Patches from Natural images
![Page 73: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/73.jpg)
83
Inference
• We learned a model• We can use it for inference problems
– Corrupted information– Missing information
• Exact inference – Loopy BP • Approximate inference - gradient based
optimization
![Page 74: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/74.jpg)
84
Belief Propagation
• Observed data is incorporated to model byiy i
ix
iy
![Page 75: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/75.jpg)
85
Belief Propagation
Message passing Algorithm
• Exact only on tree MRFs • Efficient only on pairwise MRFs
![Page 76: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/76.jpg)
86
Alternative by Roth and Black
• Reminder:
• Approximate inference by gradient-based optimization :
• Advantage: Low computational cost• Drawback: only local minimum if not convex
= argmin ( ( | )) ( ( ))MAPI
I Log P I I Log P I
Uncertainty \Noise model Learned model
( 1) ( ) , ( , )t tII I I I E I I
![Page 77: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/77.jpg)
87
Partition function
=> No need to estimate partition function
• We get:
( , ) 1
arg min ( ( | )) * ( , );n
i iI x y i
Log P I I F I x y
( , ) 1
argmin ( ( | )) log( ( , )) * ( , );n
i i i iI x y i
Log P I I Z F F I x y
X
(Doesn’t depend on )
I
![Page 78: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/78.jpg)
88
The gradient step
( ) ( )
( , ) 1 1
* ( , ); * '( * ; )n N
i iI i i i
x y i i
F I x y F F I
( )iF( )iF
• How to derivate the second term?• By a mathematical “trick” we get:
![Page 79: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/79.jpg)
89
• Assume Gaussian noise
• So the Gradient step is:
De-noising
2
2
( | ) ( )1( ( | )) ( )
2
nP I I P I I
Log P I I I I
( ) ( )2
1
1 ( ) * '( * ; )N
i ii
i
I I I F F I
![Page 80: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/80.jpg)
90
Results
![Page 81: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/81.jpg)
91
Results
![Page 82: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/82.jpg)
92
Results
Original Noisy(20.29dB)
FOE(28.72dB)
Poritilla (Wavelets)(28.9dB)
Non-local means
(28.21dB)
StandardNon-Linear diffusion (27.18dB)
State of the art
Generalprior
![Page 83: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/83.jpg)
93
Results on Berkeley databaseWiener filter
Non-Linear diffusion FOE
Poritilla1Poritilla2
Out
put P
SNR
Low noise
High noise
Input PSNRLow noise
High noise
Input PSNR
![Page 84: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/84.jpg)
94
How many 3x3 filters to take?
Number of filters
Size of filter – 3X3Performance start saturating when we reach 8 filters
![Page 85: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/85.jpg)
95
Dependence on size and shape of clique
What is the best filter?
![Page 86: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/86.jpg)
97
Inpainting - Reminder
Y AX n
Y X
Problem: pixels outside mask can change
Solution: constraint them
![Page 87: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/87.jpg)
Inpainting
• Assume pixels outside mask M don’t change
• So the gradient step is: ( ) ( )
1
* '( * ; )N
i ii
i
I M F F I
Advanced Topics In Computer
Vision CourseSpring 2010
Advanced Topics In Computer
Vision CourseSpring 2010
0-1 Mask Image we want to inpaint98
![Page 88: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/88.jpg)
99
Results
![Page 89: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/89.jpg)
100
Results
![Page 90: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/90.jpg)
101
ResultsFOE Bertalmio
FOE Bertalmio
PSNR 29.06dB 27.56dB
SSIM 0.9371 0.9167
![Page 91: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/91.jpg)
102
Pro’s and Con’s
• Perform well on narrow straws or small holes (even if they cover most of the image)
• Isn’t able to fill large holes• Isn’t designed to handle textures
![Page 92: Statistics of natural images](https://reader036.vdocuments.us/reader036/viewer/2022081507/5681666a550346895dda0119/html5/thumbnails/92.jpg)
103
Thank you for Listening…