A Comparative Study of Wavelets and Adaptively Learned Dictionary in Compressive Image Sensing

Zhenghua Zou, Xinji Liu, Shu-Tao Xia Department of computer science, Graduate school at Shenzhen, Tsinghua University, Shenzhen, China

sparkling.zou@gmail.com, jstxlxj@gmail.com, xiast@sz.tsinghua.edu.cn

Abstract—The choice of a dictionary for sparse representation is a crucial step in compressive sensing. Wavelets are very commonly used sparse basis, and K-SVD is a dictionary learning algorithm having shown its potential in sparse representation. In this paper, we combine K-SVD and compressive sensing in image sampling, and compare the performance of K-SVD dictionary as sparse basis to Daubechies wavelets. A series of tests are done on clean and noisy images at different sampling rate, results show that K-SVD can sparsely represent images very effectively, and performs much better in compressive image sensing at low sampling rate than Daubechies wavelets do.

Keywords-compressive sensing; wavelets; learned dictionary; K-SVD; image denoising; overlapped image patches; sparsity;

I. Introduction Unlike traditional sampling theory, compressive sensing

(CS) provides a fundamentally new approach to data acquisition which overcomes the common wisdom that the sampling rate must be twice the highest frequency. By compressive sensing, certain signals or images can be recovered from what was previously believed highly incomplete measurements. CS relies on two principles: sparsity, which pertains to the signals of interest, and incoherence, which pertains to the sensing modality [3] [11].

Sparsity expresses the idea that many types of real-world signals and images have a sparse expansion in terms of a suitable basis, for instance a wavelet expansion. Mathematically speaking, consider a vector f ∈ R� , we expand it in an orthonormal basis Ψ = [ψ� ψ�…ψ�] as follows.

f(t) = ∑ xψ(t)�� (1)

Where x is the coefficient vector, x = ⟨f, ψ⟩. It will be convenient to express f as x. We call the signal f k-sparse if the coefficient vector has at most k nonzero entries [10].

Sparse coding has been successfully applied to a variety of problems in computer vision and image analysis, including image compression [6], image denoising [2] [4], image restoration [11], and compressive sensing [13] [15].

Finding a sparse representation for a certain signal involves the choice of a dictionary, which is the set of atoms used to decompose the signal. Applying compressive sensing to image sampling, the first problem is to find a convenient transformation or basis which will be employed to get the

sparse representation of image. Many researchers use wavelet in compressive sensing, and a lot of work has been done [2] [5] [19]. Because of simplicity and high efficiency, wavelets has been widely used in astronomy, acoustics, nuclear engineering, sub-band coding, signal and image processing [6], neurophysiology [7], music, magnetic resonance optics [20], fractals, turbulence, earthquake prediction, radar, human vision [2].

The K-SVD algorithm was introduced by [11] as a method for sparse signal representation. The K-SVD dictionary training algorithm is an iterative method that alternates between sparse coding of the examples based on the current dictionary and an update process for the dictionary atoms so as to better fit the data. K-SVD has been applied directly to several stylized application in image processing and shows good results. In this paper, we combine compressive sensing with K-SVD dictionary learning algorithms to form a new image sampling scheme, then we test this scheme on images with varying noise level to see whether it performs better than wavelet based compressive image sensing.

The rest of the paper is organized as follows: Section describes the theoretical background of wavelets, K-SVD, and compressive sensing; Section makes a detailed description of compressive image sensing scheme based on wavelets and K-SVD; Section shows the experiment results, and section includes the conclusions.

II. Background and Prior Work

A. Compressive Sensing The basic idea of CS is that, when the image of interest is

very sparse or highly compressible on some basis, relatively few well-chosen observations suffice to reconstruct the most significant nonzero components [20].

Considering that a signal f is sparse on some basis , f =x, CS directly acquires a condensed representation using

M<N linear measurements.

= �� = ��� (2)

In the measurement process, ϕ does not depend on the signal f. We can recover this signal f by solving (3).

min ‖x‖� s. t y = ϕψx (3)

Solving (3) is a NP problem. There are two practical and tractable alternatives to (3): greedy algorithms and convex

___________________________________ 978-1-4673-2197-6/12/$31.00 ©2012 IEEE

ICSP2012 Proceedings

relaxation [10]. BP is a classical convex relaxation problem; in BP, the l1 minimization approach considers the solution of (4).

min ‖�‖� �. � = ��� (4)

In presence of noise, we can modify (4) to include noise allowance [8] [21].

min ‖�‖� �. � ‖� − ���‖ ≤ � (5)

With compressive sensing, we can exactly reconstruct k-sparse vectors and closely approximate compressible vectors stably with high probability using M ≥ O(Klog ��

!")

random measurements [9] [12].

Compressive sensing has been successfully applied in many areas such as audio and video signal compression,

B. The Wavelet Transform Wavelets are mathematical functions that cut up data into

different frequency components, then study each component with a resolution matched to its scale [1]. Compared to traditional Fourier transform, Wavelets have great advantages in analyzing physical situations where the signal contains spikes and discontinuities.

A wavelet is a function ψ L2(R) with a zero average:

∫ ψ(t)dt = 0$%&% (6)

It is normalized ||ψ|| = 1 and centered in the neighborhood of t = 0. A dictionary of time-frequency atoms is obtained by scaling by s and translating it by u:

D = 'ψ*,+(t) = �√+

ψ(/&*+

)1*∈2,+∈23

(7)

These atoms remain normalized: ||�4,5|| = 1. The wavelet transform of f L2(R) at time u and scale s is

W f (u, s) = < f, ψ*,+ >= ∫ f(t)$%&%

�√+

ψ∗(/&*+

)dt (8)

As for discrete wavelet transform, the wavelet basis can be defined as:

7(8,9)(:) = 2&?@7(2&8: − A) (9)

In (9), the scale index s indicates the wavelet’s width, and the location index l gives its position. Wavelet decomposition is implemented through a series of filtering and down sampling process. The filter is placed in a transformation matrix, which is applied to a raw data vector. The coefficients are ordered using two dominant patterns, one that works as a smoothing filter, and one pattern that works to bring out the data’s detailed information.

C. K-SVD Dictionary Learning Algorithm Dictionary training is a much more recent approach to

dictionary design, and has been strongly influenced by the latest advances in sparse representation theory and algorithms. The main advantage of trained dictionaries is that they lead to state-of-the-art results in many practical signal processing applications. The K-SVD algorithm is in some way similar to k-means algorithm [11] [17]; both k-means

algorithm and K-SVD algorithm include an iterative process of updating dictionary and sparse coding. The K-SVD algorithm updates the dictionary atom-by-atom in a simple and efficient way. In sparse coding stage, K-SVD can works with any pursuit algorithm to compute the representation vectors xi for each example yi, by approximately the solution of: i = 1, 2, . . , N, min 'C| − E�F|C�

�1 subject to C|�F|C�≤

T0 (10) In codebook update stage, K-SVD algorithm uses

Singular-Value-Decomposition (SVD) to form the core of the atom update step, and repeat it for K times (K is the number of atoms in the dictionary). For a given atom k, the quadratic term can be expressed as:

‖G − EH‖I� = JG − ∑ LP�Q

P!S� J

I

�= J(G − ∑ LP�Q

P ) −S UV

LX�YXJ

I� = JZX − LX�Q

XJI� (11

The matrix \^ stands for the error for all the N examples when the kth atom is removed. So it would be tempting to suggest the use of SVD to find alternative _^ and:9

^. Besides, in order to enforce the sparsity constraint on (11), we need to replace (11) with (12):

JZX`X − LX�QX`XJ

I� = JZX

a − LX�aXJ

I� (12)

Where hk is a matrix of size p × |qk| ( qk =rv|1 ≤ v ≤ w, �z

k(v) ≠ 0}), with ones one the (qk(v), v)th entries and zeros elsewhere.

Table 1. K-SVD Dictionary Learning Algorithm Initialization: Set the dictionary matrix D(0) Rn*K with l2 normalized columns. Set J = 1. Repeat until convergence( stopping rule):

� Sparse Coding Stage: use any algorithm to compute the representation vectors xi for each example yi, by approximating the solution of

i = 1, 2, . . , N, min 'C| ~ − E��|C��1 subject to C|��|C�

≤ T0 � Codebook Update Stage: For each column k = 1,2,…K in D(J-1),

update it by -Define the group of examples that use this atom, qk rv|1 ≤ v ≤ w, �z

k(v) ≠ 0} -Compute the overall representation error matrix, Ek, by

�^ = � − � _�:��

�UV

-Restrict Ek by choosing only the columns corresponding to qk, and obtain �^

�. -Apply SVD decomposition �^

� = U VT. Choose the updated dictionary column dk to be the first column of U. Update the coefficient vector :�

^ to be the first column of V multiplied by (1,1).

� Set J = J + 1. The K-SVD algorithm has been successfully applied to

sparsify signals and shown advantages over wavelets in image denoising [4], inpainting [11], and compression [14].

III. IMPLEMENTATION DETAILS

A. Compressive image sensing using wavelets The input image is “Lena” with size 256*256, if we

directly sample this image with sampling rate 0.5, the size of the sensing matrix should be 131072*262144, which is too big a matrix to run on a pc. Our strategy is to split the image into patches with size 8*8. We sense an image patch each time until we finish sampling the whole image. But splitting

image into patches could lead to obvious edges between the adjacent patches. To remove these edges, we split the image into patches with overlaps.

We choose random Bernoulli matrix as the sensing matrix as is shown in (13).

Φ,S = �+ �

√� with possibility 0.5

− �√�

with possibility 0.5 (13)

We use Daubechies-4 as sparsity basis. Daubechieswavelets are implemented by “UVi_Wave 3.0” toolbox [22]. Considering noise in the image, we use LASSO to reconstruct each image patch; LASSO is Implemented in sparselab [23].

After reconstructing each image, we average these overlapped patches, so that the edge effect between patches can be effectively reduced.

Fig.1 Daubechies wavelets based compressive image sensing scheme

B. Compressive image sensing using K-SVD In K-SVD based compressive image sensing, the input

image is also split into patches with overlaps. We used these patches to train an over-complete dictionary D, which acts as sparsity basis. The sensing matrix and reconstruction algorithm used in K-SVD based compressive image sensing is just the same as wavelet based compressive image sensing scheme. The K-SVD algorithm is implemented in KSVD matlab Toolbox [24]. The whole process is shown in Fig.2.

Fig.2 K-SVD based compressive image sensing scheme

C. Reconstruction performance comparison We use these two schemes to compressive sampling four

input images: clean 256*256 Lena image, noisy 256*256 Lena (polluted with Gaussian noise) images with PSNR 22.05, 19.58, 16.08 respectively. For each input image, we sample it at varying sampling rates from 0.1 to 0.9; the final

reconstruction performance is valued by PSNR, as is shown in (14).

PSNR = 20 ∗ log ( ���√���

)

MSE = ∑ (��& ��)������� ��¡¢

I£¤¥¦+§¦ (14)

IV. EXPERIMENT RESULTS

Fig.3: (a) clean 256*256 Lena image; (b) reconstructed image using

Daubechies wavelets with sampling rate 0.4; (c) reconstructed image using K-SVD with sampling rate 0.4.

Fig.4: (a) input noisy image with noise level = 30, PSNR = 18.57; (b)

Reconstructed image using Daubechies wavelets at sampling rate 0.4; PSNR = 22.00; (c) reconstructed image using K-SVD at sampling rate 0.4; PSNR = 24.71.

In this section, some experiments are carried out to demonstrate the performance of the Daubechies wavelet based compressive sensing and K-SVD based compressive sensing. The clean 256*256 Lena image is selected as the input image as is shown in Fig.3(a), we sample the clean Lena image at the sampling rate 0.4 and reconstruct it using Daubechies wavelets as sparsity basis, the reconstruct result is shown in Fig.3(b). Fig.3(c) is the reconstructed image using K-SVD as sparsity basis. From naked eye view, both images are of high quality, but Fig.3(c) is better reconstructed in details than Fig.3 (b).

In Fig.4, the noisy 256*256 image (polluted with Gaussian noise, PSNR = 18.57) is selected as the input image as is shown in Fig.4 (a). Fig.4 (b) and Fig.4(c) is the reconstructed images using Daubechies wavelet and K-SVD as sparsity basis separately, the sampling rates are both 0.4. Both reconstructed images have higher PSNR than input noisy image. The reconstruction process has a smoothing effect on the noisy Lena image. We can tell from the reconstruction result that K-SVD has a better performance than Daubechies wavelets when applied in compressive image sensing.

We sample images with varying noise level at sampling rate from 0.1 to 0.9, and reconstruct these images using Daubechies wavelets and K-SVD as sparsity basis separately; the PSNR of the reconstructed images are shown in Fig.5, Fig.6, Fig.7, and Fig.8. When dealing with clean image, compressive image sensing using K-SVD as sparsity basis achieves better performance, as is shown Fig.5. When

dealing with noisy image, K-SVD has better performance than Daubechies wavelets at low sampling rates.

The reconstructed images have higher PSNR than the input noisy image, this is partly because the “compressive” sampling process ignores some of the Gaussian noise. Another reason is that, after reconstruction, we get sparse coefficients of the image, and put a soft thresh on this coefficients, which is very similar to wavelets denoising.

Fig.5: reconstruction results comparison using Daubechies wavelets and K-SVD on clean image at varying sampling rates.

Fig.6: reconstruction results comparison using Daubechies wavelets and K-SVD, the input 256*256 Lena image is polluted with Gaussian noise and it’s PSNR = 22.05

Fig.7: reconstruction results comparison using Daubechies wavelets and K-SVD, the input 256*256 Lena image is polluted with Gaussian noise and it’s PSNR = 18.58.

Fig.8: reconstruction results comparison using Daubechies wavelets and K-SVD, the input 256*256 Lena image is polluted with Gaussian noise and it’s PSNR = 16.08.

V. CONCLUSIONS The performances of compressive image sensing using

Daubechies wavelets and K-SVD learned dictionary as sparsity basis separately are compared in this paper, the reconstruction results show that, both Daubechies wavelets and K-SVD learned dictionary can effectively sparsify image patches. Using K-SVD learned dictionary as sparsity basis performs better in compressive sensing when dealing with clean image. As for noisy image, using Daubechies wavelets in the reconstruction process performs better when the sampling rate are high (>=0.6), and both compressive sensing schemes have a smoothing effect on noisy image, thus can be used to denoise image.

This paper is an initial attempt to combine K-SVD dictionary learning algorithm with compressive sensing, we compare K-SVD trained dictionary to Daubechies wavelets, and find that K-SVD can more effectively sparsify image patches as well as denoise noisy image. Future work includes using compressive sensing combined with K-SVD trained dictionary or other proper dictionaries to denoise image effectively.

VI. ACKNOWLEDGMENT This work is supported by the NSFC project (60972011),

the Research Fund for the Doctoral Program of Higher Education of China (20100002110033), and the open research fund of National Mobile Communications Research Laboratory of Southeast University (2011D11).

References [1] Mallat, S.G., A wavelet tour of signal processing. Academic Pr,1999 [2] Starck, J.L., F. Murtagh and J.M. Fadili, Sparse image and signal

processing: wavelets, curvelets, morphological diversity. Cambridge Univ Pr,2010.

[3] Candes, E.J., "Compressive sampling". Proceedings on the International Congress of Mathematicians: Madrid, August 22-30, 2006: invited lectures

[4] Elad, M. and M. Aharon. "Image denoising via learned dictionaries and sparse representation". 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), vol. 1, pp.895-900, 2006.

[5] Duarte, M.F., M.B. Wakin and R.G. Baraniuk. "Wavelet-domain compressive signal reconstruction using a hidden Markov tree model".

IEEE International Conferecnce on Acoustics, Speech and Signal Processing, pp.5137 - 5140, 2008.

[6] Antonini, M., et al., Image coding using wavelet transform. IEEE Transactions on Image Processing, vol.1, no.4, pp. 205-220, 1992.

[7] Graps, A., An introduction to wavelets. IEEE Computational Science & Engineering, vol.2, no.2, pp. 50-61, 1995.

[8] Tsaig, Y. and D.L. Donoho,"Extensions of compressed sensing. Signal processing, vol.86, no.3, pp. 549-571. Oct. 2004.

[9] Donoho, D.L., "Compressed sensing". IEEE Transactions on Information Theory, vol. 52, no. 4, pp. 1289-1306, 2006.

[10] Fornasier, M. and H. Rauhut, " References Compressive sensing". Handbook of Mathematical Methods in Imaging,

vol. 1, pp. 187-229, 2010. [11] Aharon, M., M. Elad and A. Bruckstein, "K-SVD: An Algorithm for

Designing Overcomplete Dictionaries for Sparse Representation". IEEE Transactions on Signal Processing, vol. 54, no. 11, pp. 4311-4322, 2006.

[12] Candes, E.J. and M.B. Wakin, "An Introduction To Compressive Sampling". IEEE Signal Processing Magazine , vol. 25, no. 2, pp. 21-30, 2008.

[13] Rauhut, H., K. Schnass and P. Vandergheynst, "Compressed sensing and redundant dictionaries". IEEE Transactions on Information Theory, vol. 54, no. 5, p p. 2210-2219, 2008.

[14] Bryt, O. and M. Elad, "Compression of facial images using the K-SVD algorithm". Journal of Visual Communication and Image Representation, vol. 19, no. 4, pp. 270-282, 2008.

[15] Romberg, J., "Imaging via compressive sampling". IEEE Signal Processing Magazine, vol. 25, no. 2, pp. 14-20, 2008.

[16] Chartrand, R., Baraniuk R. G., Eldar Y. C., Figueiredo A. T., Tanner J., "Introduction to the Issue on Compressive Sensing". IEEE Journal of Selected Topics in Signal Processing, vol. 4, no. 2, pp. 241-243, 2010.

[17] Rubinstein, R., A.M. Bruckstein and M. Elad, "Dictionaries for sparse representation modeling". IEEE Proceedings, vol.98, no.6, pp. 1045-1057, 2010.

[18] Rauhut, H., "Compressive sensing and structured random matrices". Theoretical Foundations and Numerical Methods for Sparse Recovery, vol.9, pp. 1-92, 2010.

[19] He, L. and L. Carin, "Exploiting structure in wavelet-based Bayesian compressive sen1sing". IEEE Transactions on Signal Processing, vol. 57, no.9, pp. 3488-3497, 2009.

[20] Willett, R.M., R.F. Marcia and J.M. Nichols, “Compressed sensing for practical optical imaging systems: a tutorial”. Optical Engineering, vol.50, pp. 072601, 2011.

[21] Donoho, D.L., M. Elad and V.N. Temlyakov, “Stable recovery of sparse overcomplete representations in the presence of noise”. IEEE Transactions on Information Theory, vol.52, no.1, pp. 6-18, 2006.

[22] Toolbox UVi-wave version 3.0, available at http://www.control.auc. dk/~alc/html/uvi_wave_toolbox.html

[23] Toolbox Sparselab 2.1, available at http://sparselab.stanford.edu/ [24] K-SVD matlab toolbox, availabl at http://www.cs.technion.ac.il/~elad

/software/