![Page 1: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/1.jpg)
Estimating the Likelihood of Statistical Models
of Natural Image PatchesDaniel Zoran
ICNC – The Hebrew University of Jerusalem
Advisor: Yair Weiss
CifAR NCAP Summer School
![Page 2: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/2.jpg)
Natural Image Statistics
• Natural scenes and images exhibit very distinctive statistics
• A lot of research has been made in this field since the 1950s
• Important in image processing, computer vision, computational neuroscience and more…
![Page 3: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/3.jpg)
Natural Image Statistics Properties
• The space of all possible images is huge– For a 256 gray levels, NxN sized matrix, there are
possible images– Natural Images occupy a tiny fraction of this space
• Some statistical properties of natural images:– Translation invariance– Power law spectrum – – Scale invariance– Non-Gaussianity of marginal statistics - (more on
that later)
2
256N
2( ~A
Pw
![Page 4: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/4.jpg)
Estimating the Likelihood of different statistical models• During the years, a lot of models for natural
image distributions have been proposed• It is hard to test the validity of such models,
especially when comparing one model to the other
• A step towards this – estimating the (log) likelihood of a given model and comparing the results with other models
![Page 5: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/5.jpg)
Estimating the Likelihood of different models• Variable sized patches were extracted from
natural images• Different models assumed• A training set was used to estimate various
parameters of the model• Likelihood was calculated over a test set• 5000 patches in each set• Source images are mostly JPEGs from a Panasonic
digital camera, portraying outdoor scenes• Also tested on standard images (Lena, Boat and
Barbara – PNG format)
![Page 6: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/6.jpg)
![Page 7: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/7.jpg)
The models – 1D Gaussian• A 1D Gaussian distribution for every pixel
– Mean and Variance estimated directly from the sample
– The likelihood of an image x under this model is:
– Where:
• This model captures nothing about natural images
2222
2
1( | )
2
i i
i
x
i i
L e
x
1 pi i
p
xP
22 1
1p
i i ip
xP
![Page 8: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/8.jpg)
Results – 1D Gaussian
Patch SizeTest Set 1 (JPG)Test Set 2
(JPG)Test Set 3 (PNG)Test Set 4
(Noise)
10x1020.3-19.618.4-26.9
12x1229.4-28.026.2-38.8
14x1441.4-38.435.8-54.5
16x1652.1-44.647.7-70.5
18x1868.3-54.760.5-89.1
20x2084.5-6772.9-106.4
![Page 9: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/9.jpg)
The models – Multidimensional Gaussian with PCA• Using the covariance matrix, rotate the images in
the image space towards directions of maximum variance (PCA)
• A Multidimensional Gaussian distribution for the components:
• Where the covariance matrix is estimated from the training set:
• This captures the Power-Law spectrum property
/2 1/2
1( | 0 ) exp( )
2NL
T -1x Σ x Σ x
Σ
Txx
![Page 10: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/10.jpg)
Results – Multidimensional Gaussian
Patch SizeTest Set 1 (JPG)Test Set 2
(JPG)Test Set 3 (PNG)Test Set 4
(Noise)
10x1043840046526946
12x1263157769640345
14x1488679294557808
16x1611771039128579020
18x18150213341661104777
20x20191617022007135830
![Page 11: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/11.jpg)
The models – Gaussian Mixture Model with PCA• Using the same rotation scheme (PCA), now
assume a Gaussian Mixture Model for the marginal filter response distributions
• Under this model:
• Where W’s rows are the eigenvectors of the covariance matrix
• The GMM parameters were found using EM• This captures both the Power-Law spectrum and
the sparseness properties
2 2
1
( | 0 , ) ( | 0, )K
k i kki
L c N y
x σ c y = Wx
![Page 12: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/12.jpg)
Results – GMM with PCA
Patch SizeTest Set 1 (JPG)Test Set 2
(JPG)Test Set 3 (PNG)Test Set 4
(Noise)
10x10200210181-4852
12x12285295244-8138
14x14376400312
16x16472511404
18x18595646488
20x20698767557
![Page 13: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/13.jpg)
The models – Generalized Gaussian with PCA• Finally, instead of using a GMM, we now use a
Generalized Gaussian• This has the advantage of having less
parameters, while still capturing Sparseness:
• Parameters were obtained directly from the training set
( | 0 , ) ( , ) exp( )ii i
i i
L A
Wxx σ α
![Page 14: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/14.jpg)
Results – Generalized Gaussian with PCA
Patch SizeTest Set 1 (JPG)Test Set 2
(JPG)Test Set 3 (PNG)Test Set 4
(Noise)
10x10208212191-1894
12x12291305273-3099
14x14386411356-5242
16x16497529456-7875
18x18615663543-11966
20x20739796668-16783
![Page 15: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/15.jpg)
Mean Log Likelihood - 12x12 patches
-500
-300
-100
100
300
500
700
900
Noise Natural Patches
1D Gaussian
PCA-MDGaussian
PCA-GMM
PCA-GG
![Page 16: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/16.jpg)
Mean Log Likelihood - 18x18 patches
-500
0
500
1000
1500
Noise Natural Patches
1D Gaussian
PCA-MDGaussian
PCA-GMM
PCA-GG
![Page 17: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/17.jpg)
The GG shape parameter
• During the analysis of the data we have encountered a strange phenomena
• Marginal distributions get wider as we go measure higher frequency filter responses
• This is not due to increase in variance (which drops as we go to high frequencies)
• We modeled this using the shape parameter obtained from the samples
![Page 18: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/18.jpg)
Shape parameter for test set 1
![Page 19: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/19.jpg)
Shape parameter for test set 2
![Page 20: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/20.jpg)
Shape parameter for test set 3
![Page 21: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/21.jpg)
Shape parameter for test set 4 - PNG
![Page 22: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/22.jpg)
Shape parameter for noise test set
![Page 23: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/23.jpg)
Conclusion
• This is (very) early work, still in progress• A lot of things left to do:
– Try more models and filter (ICA is in progress)– Actually compare the different models– Try to make some sense out of the shape of the
distributions– Look into higher order dependencies and
correlations
• A lot more…
![Page 24: Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR](https://reader035.vdocuments.us/reader035/viewer/2022081603/56649f115503460f94c242be/html5/thumbnails/24.jpg)
Thank you!
Questions?