![Page 1: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/1.jpg)
NlogN Entropy Optimization
Sarit Shwartz
Yoav Y. Schechner
Michael Zibulevsky
Sponsors: ISF, Dvorah Foundation
1
![Page 2: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/2.jpg)
Kernel Estimators: Parzen Windows
Data
True PDF
Estimated PDF
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
49
![Page 3: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/3.jpg)
Previous Work
• Parametric PDF:Hyvärinen 98, Bell; Sejnowski 95, Pham; Garrat 97.
• Cumulants:Cardoso ; Souloumiac 93.
Not accurate
![Page 4: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/4.jpg)
• Order statistics:Vasicek 76, Learned-Miller; Fisher 03.
• KD trees:Gray; Moore 03.
Previous Work
Not differentiable
![Page 5: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/5.jpg)
Entropy Estimation
• Kernel Estimators: reduced complexityPham, 03, .Erdogmus; Principe; Hild, 03, Morejon; Principe 04,
Schraudolph 04, (Stochastic gradient).
![Page 6: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/6.jpg)
Source Range: Continuous
![Page 7: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/7.jpg)
Parzen Windows Estimator
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
50
![Page 8: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/8.jpg)
• Minimization of Mutual Information
• Differentiable
• Computationally efficient
- Currently O( K N )
Independent Component Analysis
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
online code
(see website)
51
2 2
![Page 9: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/9.jpg)
Convolution
Sampling
Parzen Windows as a Convolution
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
Wish it was … Discrete convolution
52
![Page 10: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/10.jpg)
Efficient Kernel Estimator
A. Samples of
estimated sourcesA
PDF estimation Fan; Marron 94, Silverman 82.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
53
![Page 11: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/11.jpg)
A. Samples of
estimated sources
B. Interpolation to
uniform grid
(histogram)
A
B
Efficient Kernel Estimator
PDF estimation Fan; Marron 94, Silverman 82.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
53
![Page 12: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/12.jpg)
C
• Samples of estimated sources
• Interpolation to uniform grid(histogram)
• Discrete convolution with Parzen window
A
B
PDF estimation Fan; Marron 94, Silverman 82.
Efficient Kernel Estimator
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
53
![Page 13: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/13.jpg)
D
Efficient Entropy Estimator
C
• Interpolation to original values
A
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
54
![Page 14: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/14.jpg)
Can it be Used for Optimization?
Wseparate
• Iterations exploiting derivatives of .
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
55
![Page 15: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/15.jpg)
Can it be Used for Optimization?
Wseparate
• Binning fluctuations of .
• Fluctuations amplified by differentiation.
• Fluctuations slow convergence, false minima.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
56
![Page 16: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/16.jpg)
Function
Quantized function
Quantization and Optimization
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
57
![Page 17: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/17.jpg)
Function
Quantized function
Function with a quantized derivative
Quantization and Optimization
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
57
![Page 18: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/18.jpg)
Analytic Entropy Gradient
Accurate derivative
Efficient calculation
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
![Page 19: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/19.jpg)
Complexity
Analytic Entropy Gradient
K- number of sources, N-data length.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
![Page 20: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/20.jpg)
Entropy Gradient by Convolutions
Convolution
Convolution
Convolution
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
58
![Page 21: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/21.jpg)
• Calculation of using convolutions.
• Approximation of convolutions with
complexity.
• Distinct quantization of the derivative.
Not differentiation of a quantized
function.
Entropy Gradient by Convolutions
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
59
![Page 22: NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah Foundation 1](https://reader030.vdocuments.us/reader030/viewer/2022032522/56649d615503460f94a42549/html5/thumbnails/22.jpg)
K=6 random sources, N= 3000 samples.
AlgorithmSignal to Interference ratio [dB]
Time
Basic Non-param ICA
18 4760 min.
Our algorithm22 31.2 min.
Jade7 40.2 sec.
Infomax8 41.6 sec.
Fast ICA5 31.9 sec.
Super ICA performance
Parametric
algorithms.
Non-parametric
algorithms.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
60