NNSA Office of Proliferation Detection Funding Opportunity Number: DE-PS52-

09NA29330 CFDA Number: 81.113

Topic 1: Remote Sensing – Hardware, Software/Algorithms, and System Development

Subtopic 1d: Novel Remote Sensing Approaches

Project Title: Compressive Sensing and Super-Resolution of Passive Millimeter Wave Images

Project Director/PI: Aggelos K. Katsaggelos, Professor Co-PI: Thrasyvoulos N. Pappas, Associate Professor

Address: Department of Electrical Engineering and Computer Science

Northwestern University Evanston, IL 60208-3118

Emails: mailto:aggk@eecs.northwestern.edu pappas@eecs.northwestern.edu

Voice: (847) 491-7164, (847) 467-1243

August 25, 2009

1. Introduction With the wider availability of nuclear technology and rising world tensions, nuclear non-proliferation has taken a center stage in US national security. However, while the potential threats have increased, so have our possibilities for detection and prevention. The early detection of nuclear threats, both intentional (weapon development) and unintentional (accidents) can be accomplished by novel remote sensing systems.

Passive millimeter-wave imaging (PMMWI) provides a broad area of “search and cueing” capabilities to detect undeclared nuclear facilities based on such observables as thermal signatures, building structures, plumes, and traffic near the facilities. It offers various advantages over optical imaging, such as, long-range (>1km) detection under adverse environments, including smoky, dusty, and cloudy conditions, during both day and night times.

There are a number of open problems that need to be addressed with any PMMWI system. One such critical problem is the extended acquisition time. We need to build PMMWI systems that are as fast as possible in acquiring the scene, especially if we are to capture some of the dynamics in the scene. A second problem related to the first one is the design of PMMWI systems that will be able to image close to the Rayleigh resolution limit, which is determined by the characteristics of the lens. A third problem is the development of signal post-processing techniques that will allow us to super-resolve the data, i.e., image below the resolution limit.

The focus of this proposal is to address all three of the above issues for single antenna/pixel PMMWI systems. We propose to address the first two problems mentioned above by extending the design of a PMMWI system through the use of coded masks and the application of compressive sensing algorithms. We also propose to address the third problem mentioned above by developing non-linear and spatially varying super-resolution algorithms. The development of both the compressive sensing reconstruction and super-resolution algorithms will be based on recovery algorithms that have been developed at Northwestern over the period of many years. Such algorithms are based on Bayesian inference through appropriate modeling of the scene and the degradation process. They rely exclusively on the data to determine all the unknown parameters and the original scene, i.e., they do not require any human intervention.

The proposed algorithmic developments will be in close collaboration with the Argonne National Laboratory where a novel single-pixel PMMWI system is developed, and will nicely complement their work. The derived results can be applied to imaging at other wavelengths, such as infrared. The proposed technology therefore can have wide applicability to various programs within NNSA.

This proposal will primarily address Topic 1d Novel Remote Sensing Approaches. However, the results of the proposed research will also contribute to other research topics, such as Topic 2 (Simulation, Modeling, and Algorithms) and Topic 5 (Global Safeguards).

2. Project Objectives The objectives of the proposed research are to develop novel single pixel PMMWI systems and algorithms for the compressed sensing of images and their post-processing for increasing their resolution. These two tasks are tightly intertwined, as the specific interpretation of the

2

acquired data through advanced signal processing techniques, allows for the design of sophisticated PMMWI structures of improved efficiency in terms of acquisition times and resolution of the imaged scene. The developed hardware and software will greatly advance the state of the art in single pixel PMMWI in creating inexpensive portable systems of high efficiency and accuracy. The proposed work represents a truly collaborative effort between Northwestern and the Argonne National Labs (ANL). Northwestern will provide primarily the algorithmic know-how while ANL will provide the required hardware implementation, the resulting data and will validate the results. The advanced signal processing techniques to be developed will also find applications with other imaging modalities. Algorithms developed in this study will have a broad applicability to other imaging programs within NNSA.

3. Discussion of Merit Review Criteria In this section, we describe the basic science and methodology for the proposed research, and address each of the merit review criteria.

3.1. Mission Relevance The goal of the proposed research is to develop the signal processing techniques that will result in improved single pixel PMMWI systems. Such systems will have shorter acquisition times and higher resolution than existing ones. They will therefore be able to provide more efficient and accurate detection of thermal signatures, building structures, and plumes, leading to the early detection of nuclear threats, both intentional (weapon development) and unintentional (accidents). The advanced signal processing techniques to be developed will also be applicable to other imaging modalities (wavelengths and structures).

3.2 Overall Scientific and Technical Merit In this section we discuss the current state of science and technology and provide the technical details of the proposed approach.

3.2.1 Passive Millimeter Waves (PMMW)

3.2.1.1 Physical Principles

Any object with nonzero temperature emits electromagnetic waves in a broad spectral range. The spectral energy density of a blackbody radiation is given by Planck’s law as

, where c is the speed of light, h is Planck’s constant, k is

Boltzmann’s constant, T is temperature, and λ is the wavelength of radiation. Although thermal radiation is emitted over a very broad spectrum, remote sensing with passive

3

millimeter waves (PMMW) offers the advantage of low atmospheric attenuation because of the long radiation wavelength.

Millimeter-waves are therefore much more effective (lower attenuation) than infrared in poor weather conditions such as fog, clouds, and snow. The amount of radiation emitted in the millimeter-wave range is times smaller than the amount emitted in the infrared range. However, current millimeter-wave receivers have at least

times better noise performance than infrared detectors and the temperature contrast

recovers the remaining . This makes millimeter-wave imaging comparable in performance with current infrared systems.

3.2.1.2 PMMW Imagers

PMMW imagers generally consist of an imaging lens or mirror that focuses the radiation from a distant object into the antenna of a radiometer. Two types of imaging systems are commonly employed: a focal plane imager and a single-element or single-pixel scanning imager. In a focal plane imager, the image of a distant target is formed at the focal plane of a lens, and an array of receiver antenna elements at the focal plane sample the brightness levels and produce the image. In a scanning single-pixel imager, the image of the target is produced by mechanically scanning the antenna-lens combination. While the focal plane system produces the entire image in one acquisition with parallel elements, it suffers from (a) poor signal to noise ratio (SNR) because only part of the radiation is coupled to each antenna element and (b) poor spatial resolution because of the minimum required antenna aperture (~ one wavelength) for efficient coupling of the radiation intensity. On the other hand, the single-pixel imager receives the full intensity of the focused radiation from different scanned portions of the target (hence with higher SNR), but the acquisition time becomes large since N acquisitions are needed for an N pixel image. Furthermore, mechanically scanning the radiometer-lens system is unwieldy and prone to cable-wiggling noise in a field scenario.

3.2.2 Proposed PMMW Imaging System

Argonne is working on a novel single-pixel imager in which neither the lens nor the radiometer antenna is scanned thus avoiding the cable noise but a coded aperture mask is scanned at the focal plane of the lens to produce a set of coded aperture images. The image at the focal plane is then reconstructed from the coded aperture images This system combines the best of single pixel imager and focal plane imager since (a) at each mask position nearly full field intensity is received by the single-pixel receiver with high SNR and (b) the pixel resolution of the mask (the 1s and 0s) is not limited by the antenna aperture as in an antenna array thus potentially reaching the diffraction limit of the lens.

We propose to implement the PWMM imager as shown in Fig. 1. It is based on the imager that has been developed at ANL (Gopalsami:2008). The main modification and innovation is the introduction of the dynamically changing coded aperture mask placed in front of the detector. The role of the mask is to spatially modulate the input signal, which is assumed to be coming from a static scene. Each unique pattern in the transparency mask modulating the signal measured by the detector corresponds to a

4

different measurement (it is the projection of the scene onto the mask). Since the coherence length of PMMW radiation is much smaller than the wavelength, the apertures in the mask may be made much smaller than the wavelength and still be considered uncorrelated. Thus it is possible to obtain a multi-pixel high-resolution image (approaching the Rayleigh resolution limit) from a compact system, as compared to an antenna array. A two-axis controller is shown in Fig. 1, as a way to implement the utilization of the various versions of a mask type. If, for example, a Hadamard mask is used, the mask (consisting of open (logical 1) or closed (logical 0) array elements on a piece of metal) can be made larger than needed for each acquisition, so that during each acquisition only a part of it is used. The mask is positioned at the appropriate place with the use of the two-axis controller. For other implementations of the mask the controller may not be required. Various types of masks will be considered; for example, a Hadamard mask, a scrambled Hadamard mask, and random masks (using, for example, the modulation scattering technique described in [Bolomey:2001]). The team of collaborators at ANL will provide solutions and implementations of such masks.

In order to minimize the number of measurements, we propose to utilize compressive sensing, as explained in the following sections. According to it the number of measurements made is considerably smaller than the number of pixels in the reconstructed image, thus considerably reducing the integration time. We propose to develop novel and computationally efficient non-linear and spatially-varying reconstruction algorithms that will reconstruct the original image at its native resolution from the reduced number of measurements.

Finally, blurring is introduced in the acquired image due to the finite aperture of the lens. There is the need therefore to restore the acquired image in order to improve its quality and resolution. Here, prior information about the structure of the object of interest is used to obtain improved results. Although a number of techniques have been proposed in the literature for solving this problem, most such techniques are outdated and they rely on manual tuning of parameters. We propose to develop state-of the art recovery techniques for the problem at hand using a Bayesian framework. Such techniques will be non-linear and spatially varying and they will estimate all unknown parameters based on the data (i.e., no need for manual tuning). We also propose to solve the compressive sensing reconstruction problem and the restoration/super-resolution problem simultaneously, using the Bayesian framework. Summarizing, the proposed work focuses at introducing various masks and developing the reconstruction and super-resolution techniques, as depicted in Fig. 1.

5

Figure 1: Proposed PMMW Imaging System with Aperture CodingAn image acquired by the currently working single-pixel PMMWI system at ANL

is shown in Fig. 2. The image represents a circular hole on a metal plate with two thin hex wrenches hanging from the top of the plate at the front of the plate extending 3/4 of the hole diameter with increasing separation between the two wrenches from top to bottom. The hole diameter is 27 mm, the diameter of the wrenches is 1.8 mm, the separation between the wrenches is 3 mm at the top of the hole and 4.5 mm at the bottom of the hanging end. The test was designed to show the resolution limits. As the image in Fig. 2 shows, the wrenches are barely distinguishable.

5 10 15 20 25 30 35 40 45 50

5

10

15

20

25

30

35

40

45

50

Figure 2: PMMW image acquired by the ANL imaging system (Gopalsami:2008)

In the following we first briefly review the theory behind compressive sensing, followed by a brief review of the approaches that have been applied to the super-resolution problem. We then present in detail the approaches we propose to follow in developing solution for the problem at hand.

3.2.3 Review of Compressive Sensing

6

3.2.3.1 Work in the Literature

In a traditional acquisition system, all samples of the original signal are acquired (in this discussion we assume that the original scene is discrete with a native resolution). This number of signal samples can be in the order of millions, as is the case for instance with digital images. The acquisition process is followed by compression, which takes advantage of the redundancy (or the structure) in the signal to represent it in a domain where most of the signal coefficients can be discarded with little or no loss in quality. For instance, for a typical image of a natural scene in the visible wavelength range, an almost lossless approximation can be achieved with only about 5% of the frequency (e.g., wavelet, DCT) coefficients. Hence, traditional acquisition systems first acquire a huge amount of data, a significant portion of which is immediately discarded (compression). This creates an important inefficiency in many practical applications. Compressive sensing addresses this inefficiency by effectively combining the acquisition and compression processes. Traditional decoding is replaced by recovery algorithms that exploit the underlying structure of the data [Candes:2006] [Donoho:2006].

Compressive sensing (or sampling) (CS) has become a very active research area in recent years due to its interesting theoretical nature and its practical utility in a wide range of applications. Compressive sensing is a new approach to capture a wide range of signals at a rate significantly lower than the Nyquist rate. In the following we provide a brief overview of the basic principles of CS, since they will form the basis of the proposed imaging system.

Let represent the unknown signal represented as an vector in . The

signal can be represented in a basis (such as a wavelet basis) with the basis coefficients

, that is, . Moreover, is called sparse in if most of the coefficients are zero, such that can be represented in with only basis vectors with . Consider the following general linear acquisition system

(1)

where linear measurements of the original unknown signal are taken with an measurement matrix . We can also write (1) in terms of the sparse signal coefficients as

(2)

The key properties of the acquisition system in (2) are a) instead of point evaluations of the signal, the system takes inner products of the signal with the basis vectors , b)

the number of measurements is considerably smaller than the number of signal

coefficients . If , (and equivalently ) can be recovered from in a

straightforward manner by inverting . However, a reconstruction process is needed when . The central result of CS [Candes:2006][Donoho:2006] is that when the signal has

7

a sparse representation, and the measurements are incoherent, the signal can be

reconstructed from with a very high accuracy even when ( in the order of log ). The

measurement matrix can be chosen as noise-like, random matrix, which generally exhibits low-coherence with any representation basis. The savings in the number of

measurements in practice are generally around to in typical acquisition systems, but

much higher savings are achieved when the signals of interest have a highly sparse representation.

After the acquisition process, an estimate of the signal is obtained by a reconstruction algorithm. The original work on CS [Candes:2006][Donoho:2006] employed regularization based on -norms and linear programming, such that the signal is reconstructed using the following optimization problem

(3)

where denotes the -norm of a vector, reinforcing a sparse solution. Many other reconstruction algorithms have been proposed in the literature. We have also developed our own and we propose to develop appropriate ones for the problem at hand in Sec. 3.2.4. Reconstruction examples are given at the end of this section.

CS is currently a research area with very high activity, and with the proliferation of new applications the need for more advanced methods and a more thorough investigation of this new methodology is expected to grow. CS research currently advances in three major fronts: (i) the design of CS measurement matrices , (ii) the development of new and efficient reconstruction techniques, and finally (iii) the application of CS theory to novel problems and hardware implementations. The first two topics have already achieved a certain level of maturity, and many advanced methods have been developed. Currently, very high efficiency CS measurement systems have been developed with different characteristics (deterministic/non-deterministic, adaptive/non-adaptive) that can be adopted in a variety of signal acquisition applications. On the other hand, reconstruction methods span a wide range of techniques that include Matching Pursuit/Greedy, Basis Pursuit/Linear Programming, Bayesian, Iterative Thresholding, among others [SPM:2008]. Which method is selected depends on the performance and speed needs of the application of interest.

Finally, new areas where CS techniques can be applied are being continuously proposed. Some interesting examples include the single pixel camera [Duarte:2008], compressive sensing MRI [Lustig:2007], compressive coded aperture imaging [Marcia:2008)][Marcia:2009], compressed DNA microarrays [Dai:2009a], and compressive light field sensing [Babacan:2009e]. Among these applications, the single pixel camera [Duarte:2008] drew a lot of interest. This system is a prototype camera consisting of a photon detector, a digital micromirror device (DMD) and two lenses. The camera takes random linear measurements of the scene under view by randomly coding the DMD, passing the light coming from the scene through the DMD and collecting multiple samples of the same scene. These collected random measurements are then used

8

in a reconstruction algorithm to recover the original scene. Effectively, the camera enables a tradeoff between the size of the camera sensor and the acquisition time, but due to the random-coding and reconstruction in the post-processing phase, the number of measurements is significantly smaller than the number of image pixels. Clearly, the savings in the number of measurements and the acquisition time are directly coupled with the performance of the reconstruction algorithm utilized during post-processing.

Unfortunately, the original implementation of the single-pixel camera only serves as a proof-of-concept, since current consumer digital cameras with megapixel-range sensors are ubiquitous, which makes such a design not necessary for visible range imaging. On the other hand, the concept of single-pixel camera is very important for its high potential in imaging applications outside the visible range (such as MMW), since imaging at certain wavelengths with high spatial resolution is much more complicated and expensive. Therefore, this simple yet important application of CS to imaging can be of crucial importance in designing novel imaging hardware. It forms the basis of the system we propose in this work (see Fig. 1).

3.2.3.2 Work at IVPL

The Image and Video Processing Laboratory (IVPL) at Northwestern University has devoted a significant amount of effort to compressive sensing research, focusing on developing state-of-the-art CS reconstruction algorithms and designing CS hardware for novel imaging applications. In the following we will summarize some of the work developed within IVPL related to compressive sensing, some of which was presented in [Babacan:2009a] [Babacan:2009b] [Babacan:2009c] [Babacan:2009e] [Vera:2009].

Our preliminary focus has been on designing advanced CS reconstruction methods that address the shortcomings of existing methods in the literature. Many existing algorithms require that the users have a deep understanding of the specifics of the algorithms, so that their parameters can be tuned specifically to the system and signals of interest. This severely limits their application since the methods can deliver very low performance or even fail when their parameters are not chosen appropriately (an example of such a parameter is in Eq. (3)). In [Babacan:2009a] [Babacan:2009b] [Babacan:2009c], we have addressed this issue and formulated the CS reconstruction problem from a Bayesian perspective. The proposed algorithms resulting from this framework simultaneously estimate the unknown signal along with all needed algorithmic parameters, so that no user-intervention is needed and the parameters are chosen optimally based on the available data. In addition to making the use of the algorithm straightforward, this property also widens the applicability of the algorithms since the parameters are estimated optimally for each specific application.

Moreover, despite being fully automated, the proposed algorithms provide competitive and even higher reconstruction performances than state-of-the-art methods, thanks to the complex modeling utilized in these works. In [Babacan:2009a], [Babacan:2009c], we proposed the use of Laplace priors on the unknown signal to model its sparsity. Laplace priors enforce sparsity to a higher extent than many existing formulations, and also have nice mathematical properties such as resulting in convex optimizations. Using our model, we developed a constructive (greedy) algorithm designed for fast reconstruction useful in practical settings. In [Babacan:2009b], we

9

developed a novel Bayesian framework using non-convex -norms, which enforce sparsity to a higher extent, and therefore achieve a higher reconstruction performance at the expense of speed of reconstruction. We have also shown that both our developed methods include many existing methods as special cases. Finally, the proposed algorithms provide estimates of the uncertainty of the reconstructions, which can be further utilized to adapt the measurement process for more accurate reconstructions. The proposed algorithms provide higher reconstruction performance than other widely used algorithms in the literature. Some example results are shown in Fig. 3, where average reconstruction errors for a number of algorithms are shown with varying number of measurements M. Our method is denoted by Laplace, which is the method proposed in [Babacan:2009a] [Babacan:2009c]. It is clear that it provides lower reconstruction errors than other state-of-the-art and popular algorithms (BCS, BP, OMP, FAR, and GPSR) for all measurement levels.

Figure 3. Performance of the proposed algorithm in [Babacan:2009a]. Number of measurements M vs reconstruction error for a number of CS reconstruction algorithms. Our proposed algorithm is denoted by Laplace (green line).

An additional compressive sensing example on application of CS to imaging is shown in Fig. 4. A compressive acquisition system is applied to the original image shown in Fig. 4(a). The image consists of 4096 pixels, and only 2073 random measurements are collected using a random uniform spherical ensemble. These random measurements are then used with different reconstruction algorithms to estimate the original image. The reconstruction results from the FAR and FDR algorithms are shown in Figs. 4(b) and (c), and the reconstructed image provided by the proposed algorithm in [Babacan:2009a] is shown in Fig. 4(d). The parameters of the proposed algorithm are solely estimated from the measurements so the algorithm is fully-automated. It can be observed from Fig. 4 that all algorithms provide fairly good approximations, whereas the proposed algorithm provides the most faithful reconstruction among other methods.

10

(a) (b)

(c) (d)

Figure 4. Examples of reconstructed Mondrian images from [Babacan:2009a] using a multi-scale compressive sensing scheme by (a) original image, (b) StOMP with FAR thresholding (error = 0.14673, time = 19.759 s, no. of nonzero components = 1196), (c) StOMP with FDR thresholding (error = 0.1747, time = 6.48 s, no. of nonzero components = 2032), (d) the proposed approach (error = 0.14234, time = 15.982 s, no. of nonzero components = 1125).

We are currently continuing our work on the CS reconstruction focusing on improving the speed of the algorithms and on incorporating more complex models which resemble real-life signals more closely (see, for example, [Vera:2009] for some preliminary results).

We have also developed a novel imaging system at IVPL based on the CS principles, whose details will be given in Section 3.2.5, since it relates closely to the proposed system in Fig. 1.

3.2.4 Review of Super-Resolution for PMMW ImagesA passive millimeter wave (PMMW) imaging system can be mathematically described by the following equation

g(i,j) = (h * f) (i,j) + n(i,j), (4)

where f(i,j) represents the undistorted image, n(i,j) the noise (a combination of the instrumentation fluctuations and discretization noise if a continuous-discrete model is replaced by its numerical analog [Pirogov:2004]), h(i,j) the point spread function (PSF)

11

or the antenna pattern of the PMMW instrument, * the 2D convolution operator, and finally g(i,j) the observed image at the output of a radio-imaging system. The super-resolution problem, as addressed in the PMMWI literature, calls for finding an estimate of f(i,j) given the observed image g(i,j), and knowledge about h(i,j), n(i,j), and possibly f(i,j). The PSF h(i,j) of the instrument can be estimated offline using experimental methods or it can be calculated analytically using the physical model of the instrument. The solution of the problem described in Eq. (4) is generally referred to as a deconvolution problem [see, for example, Katsaggelos:1991]; if h(i,j) is also unknown it is referred to as the blind deconvolution problem (for a recent review, see [Bishop:2007]). Image recovery is the more general term applied to both deconvolution and super-resolution (as well as a number of other problems, e.g., inpainting, concealment, removal of blocking artifacts, etc), according to which the objective is to recover lost information, due to the acquisition system, processing (such as compression and subtitling), or transmission. There is also a large amount of work in the literature addressing the (geometric) super-resolution of images and video when multiple acquisitions of the scene are available with sub-pixel shifts among them (see, for example, [Katsaggelos:2007]). Such approaches could also be potentially applied to the super-resolution of PMWW images.

It is mentioned here that linear inversion techniques of Eq. (4) cannot generate frequencies in the recovered image beyond the cut-off frequency of the imaging system (Rayleigh resolution limit). It is therefore non-linear inversion techniques that could do that thus achieving super-resolution. Non-linearity may be introduced through the use of non-linear constraints (such as the use of a lower and upper bound on the values of f(i,j), --the positivity constraint is a special case of this—[Schafer:1981]), or through the appropriate modeling of f(i,j), as will be shown in the proposed research section (Section 3.2.5). Another important property of the recovery system is non-stationarity or spatial adaptivity. It is desired that the recovery process takes into account the spatial properties of f(i,j) and appropriately adapts its operations. For example, noise suppression is desirable at the flat areas of the image but not at the edges, based for example, on properties of the human visual system, that is, more specifically its masking effect (high spatial frequencies in the signal mask high frequency noise). Although spatially adaptive (or non-stationary) restoration filters have been proven to provide improved results over non-adaptive (or stationary) filters and they have therefore been utilized widely for a long time in general image recovery applications (e.g., [Banham:1997]), their application on PMWW images has not been reported yet.

The imaging system in Eq. (4) can be represented in matrix-vector form as

(5)

where , , and are N x 1 vectors constructed from their corresponding images using a

lexicographic ordering, and is the N x N system matrix representing the system

degradation ( is not square in general, but with zero padding of f(i,j) can be made

square; for the deconvolution case can be written as a block-circulant matrix).

12

A number of methods have been proposed in the PMMWI literature for the solution of Eq. (4) or (5). Relatively recent reviews of the major approaches can be found in [Lettington:2002], [Pirogov:2004]. Based on a review of this literature, by enlarge such methods do not represent the state-of-the-art approaches in solving inverse problems, such as the image deconvolution and super-resolution problems. There seems to be a lag between the development of image recovery approaches in the image processing literature and their application to the recovery of PMMW images. We propose to extend and appropriately adapt some of the recovery techniques we have developed (see, for example, [Molina:2006][Molina:2008][Babacan:2009d][Babacan:2008c][Chantas:2009]) so that they are applied to the problem at hand. We believe that improved results will be obtained for PMMW images as well.

We provide a brief review of the approaches applied towards the solution of Eq. (5) for the PMMWI application (some references are also made to approaches outside the PMMWI literature). Such approaches can be found in any image processing textbook or a review paper (e.g., [Banham:1997]).

The most straightforward approach to this problem is to use least squares estimation and select

(6)

as an estimate of the original image, where denotes the generalized inverse of a matrix. Note that (6) corresponds to the maximum likelihood approach which minimizes the likelihood of the observation . However, as is well known, this approach does not lead to useful restorations in most cases because of the observation noise and severe overfitting. Incorporating knowledge about the unknown image significantly improves the quality of the super-resolution. This knowledge is incorporated in a number of ways, such as, prior distributions within Bayesian methods, constraint functions in regularization methods, and convex sets in POCS methods. The performance of different methods generally depends on the way this knowledge about the unknown image is incorporated into the algorithm and on the strength of the enforcement of such knowledge. In the following, we review the most widely used methods.

Obtaining from and is an ill-posed inverse problem due to the existence of

noise and the low-pass characteristic of . A well known approach to address ill-posedness is regularization, i.e., a solution to the inverse problem is found according to the constrained least-squares solution

(7)

where is an operator, is the Lagrange multiplier controlling the tradeoff between fidelity to the data (first term in (7)) and smoothness of the solution (second term in (7)). The operator is in general a high-pass operator such that smoothness in the restoration is enforced. The restoration based on (7) is typically referred to as Tikhonov

13

regularization, and it can generally be implemented both in the spatial or frequency (Fourier) domains (assuming is also a block-circulant matrix). Finally, (7) corresponds

to the maximum a posteriori (MAP) estimate of .

The choice of the regularization operator and the Lagrange multiplier λ are two key factors determining the quality of the super-resolved image. The parameter λ is generally chosen specific to the application and is tuned by hand (although there are approaches allowing its automated selection [Galatsanos:1992]), but more attractive choices are available when a fully Bayesian analysis is applied (see below). For this regularization operator , different choices have been proposed in the literature. When

, where and are the correlation matrices of the noise and the signal,

respectively, then the solution in (7) corresponds to the parametric Wiener filter. Another possible choice is to use for the Laplacian operator. In this case the filter is linear; if

is computed in an iterative fashion at each iteration step, then a non-linear filter results.

Note that both (6) and (7) can be implemented in the following two ways: (i) As an one-step solution, implemented either in the spatial or the discrete-frequency domain (assuming that is a block-circulant matrix); (ii) In an iterative fashion. The general form of the iterative procedure is given by [Katsaggelos:1989], [Katsaggelos:1991b], [Katsaggelos:2009]

(8)with (9)

where represents the relaxation parameter which controls convergence and the rate of convergence. Iterative algorithms based on (8) are also called successive approximation iterations (the Jensen-VanCittert and Landweber iterations are specific cases of (8)). An iterative procedure based on (8) is proposed in [Zheng:2008a], where wavelet domain operators are utilized for to model the high-pass characteristics of the image and to enforce smoothness.

An important property of the iterative solution in (8) is that certain constraints can also be imposed on the solution during the iterations. Of the most common constraints is the positivity constraint [Schafer:1981], [Richards:1988], where the pixel intensities in the solution are constrained to be nonnegative. Another possible constraint is to limit the pixel values to a certain range [Lettington:2003]. These type of constraints can also be formulated as convex sets. The Projection onto Convex Sets (POCS) method, can then be applied to find a solution in the intersection of a number of sets, describing the fidelity to the data and desirable properties of the solution. POCS methods are similar in spirit with iterations in (8), although they are derived using a different mathematical formulation

14

[Xiao:1996] [Sundareshan:2004] [Katsaggelos:1989]. Algorithms using both POCS methods and iterations similar to (8) have also been proposed in [Zheng:2007] [Zheng:2008b].

In the presentation so far we have assumed that the degradation and the images in (5) are deterministic and only the noise represents a stochastic signal. However, they can also be treated as stochastic quantities and a super-resolution solution can be obtained using the Bayesian paradigm. In a Bayesian formulation, we first form the conditional probability distribution function (pdf) of the noise as follows

(10)

Smoothness constraints on the original image are encapsulated in image priors of the form

(11)

The parameters α and β are typically referred to as hyperparameters, and they represent the inverse variances of the original image and noise, respectively. If these parameters are assumed to be known, an estimate of the image can be obtained using the maximum a posteriori (MAP) solution

(12)

The MAP solution has the maximum likelihood (ML) solution as a special case (the prior in (11) is set to a uniform distribution). However, ML solutions are generally inferior to MAP approaches as they do not incorporate prior knowledge about the original image.

If the hyperparameters are not known, they also have to be estimated and there exist multiple methods for their estimation: 1) They can be regarded as hidden data, and be estimated using an expectation maximization (EM) method; 2) they can be integrated out from the optimization problem with appropriate choices of their corresponding priors; 3) a fully Bayesian method can be developed using sampling approaches, or a variational Bayesian analysis.

Among these, the Lucy-Richardson method has been applied to the PMMW super-resolution problem in [Richardson:1972] [Lucy:1974]. With it, a uniform image prior is adopted, and an iterative procedure is obtained in the form of (8). Another method has been proposed in [Hunt:1992], where a prior distribution modeling the scene as Poisson emitters is utilized, and a MAP approach is adopted. Finally, another probabilistic method is proposed in [Lettington:1995] where the scene intensities are modeled using a Lorentzian function.

15

3.2.5 Proposed Research Approach

3.2.5.1 System Modeling

In this proposal, we consider a coupled passive millimeter wave image acquisition/super-resolution system based on compressive sensing and image recovery principles. The proposed system has two major objectives: 1) Improving the operating speed of the PMMW imager using nonlinear compressed sensing reconstruction methods; and 2) Improving the final resolution of the images with novel super-resolution algorithms. As the complete description below will show, these two objectives can be combined in a single post-processing method. On the other hand, they can also be applied independently to specific PMMW imaging systems which in turn extends their applicability. First we provide a description of the proposed compressed PMWW imaging system and then we will provide details about the proposed super-resolution algorithm.

In the proposed approach, one performs full-field measurement radiometry, which significantly improves the operational speed and image quality over that of a single aperture scan. A coded mask is placed between the antenna and the main lens of the PMMW imager, which modulates regions of the antenna by specific amounts (see Fig. 1). The fundamental effect of the mask is that each acquired pixel is a linear combination of all pixels in the target scene. As will be shown below, this approach has many advantages over single aperture scan.

The mathematical principle behind this idea is formulated as follows. Let us assume that the resolution of the PMMW imager is N pixels for simplicity. During each acquisition i, each element j in the mask is assigned a weight which controls the amount of energy sensed in this element. Each element j corresponds to , one of the N pixels in the target scene, where is the full-image of the target scene. The utilization of the mask provides images that are linear combinations of the scene, that is, the acquired pixel value can be expressed as

(13)

Note that in single aperture scan systems, only one element is open (logical 1) at each instant i such that pixels of the image are collected one at a time. After M acquisitions (), the complete set of pixel measurements can be expressed as

(14)

which can be expressed in more compact form as

(15)

with the measurement matrix ( was used in Eq. (1)) and the observed image. At the

end of the measurement process, one needs to recover from . If M=N and the system

matrix is invertible, this process is fairly straightforward disregarding the noise, as

16

can be obtained directly using . In the presence of noise, one needs to implement reconstruction algorithms in order to avoid overfitting to noise and to obtain smooth images. On the other hand, if , the image cannot be obtained directly from the measurements and specific reconstruction algorithms have to be applied. In this proposal, our goal is to investigate both designs ( and ). Let us now demonstrate the advantages of utilizing such a system with a mask instead of a single scan design with an example.

If a Hadamard matrix is chosen for , then one can perform measurements, each with a different pattern of the Hadamard mask, corresponding to different rows in the matrix . For any single measurement a large number of the elements of the mask are open, due to the property of Hadamard matrices [Decker:1970]. (Actually, the number of open apertures decreases with the number of apertures. For example, for a 7x7 mask, the fraction of transparent area is 0.32; for a 63x63 mask, the fraction is 0.25.) This increases the signal-to-noise-ratio (SNR) in the acquired images relative to the SNR of a scanning single aperture system, which leads to a corresponding reduction in the measurement time. Since passive radiation is incoherent, interference effects between electromagnetic fields passing through different apertures in the Hadamard mask may be neglected at a first approximation, and the detected intensity is a linear superposition of intensities emerging from each aperture in the mask.

Due to the property of the Hadamard matrices, the input scene can be reconstructed numerically with a simple inversion algorithm, which constitutes one of the advantages of using the Hadamard Transform. The reconstructed image is obtained from

, where , with J a matrix of all ones and S the Hadamard generating matrix [Decker:1970]. Note that this process is a linear multiplication process, and it is very advantageous with its easy implementation and very low running times. The PMMW acquisition system using a full Hadamard-based measurement and linear reconstruction will be implemented as the first step in our proposal. Although it significantly increases the SNR and correspondingly decreases the imaging times, it can provide low-quality images in the presence of heavy measurement noise due to the linear reconstruction. Moreover, one still needs to acquire N pixels in order to obtain final images with N pixel. Compressive sensing and nonlinear reconstruction provides very attractive alternative methods that can deliver higher quality images while further reducing the acquisition times.

As stated earlier in Section 3.2.3, according to the theory of compressive sensing [Candes2006][Donoho2006], the original image can be recovered from with a very high accuracy even when provided that two conditions hold: 1) the PMMW acquisition system in (15) is an incoherent measurement system, and 2) has a sparse representation in a domain. We show next that both conditions hold for the proposed system.

Note that property 1) is a design issue. A sufficient condition for a matrix to be a compressive sensing matrix is the restricted isometry property (RIP) [Candes2005][Candes2006], which is proven to hold with a very high probability for a general class of matrices with their entries drawn from certain random probability distributions. There is a wide selection of compressive sensing matrices that can be utilized for PMMW systems,

17

among which we will consider two: 1) Uniform ensembles where the values of the matrix are drawn randomly from the interval (0,1) and the columns are normalized, and 2) Scrambled Hadamard matrices [Gan:2008] where the matrix is constructed by randomly choosing columns of a full Hadamard matrix. Note that uniform ensembles require fractional values in the mask. On the other hand, scrambled Hadamard matrices only require binary values in the mask and can be implemented very easily in practice using plastic or metal masks.

Property 2) is related to the nature of the target images. A well-known fact is that natural images can be represented sparsely in many transform domains. Typical examples include wavelets, gradients, and total-variation (TV) functions. This property is the key in many image compression, denoising and super-resolution systems. As known in the PMMW literature [Lettington:2002] [Pirogov:2004], PMMW images can also be represented sparsely in many transform domains, and they are therefore suitable for recovery within a compressive acquisition system such as the one proposed here.

By combining properties 1) and 2), it is clear that the proposed PMMW acquisition system is a valid compressive sensing system. Therefore, it will exhibit the main advantages of CS systems, that is, the target scene can be reconstructed with much fewer measurements than traditionally required. One of the main objectives of this proposal is to implement and test this system to achieve much higher acquisition speeds than traditional PMMW systems.

Note that the reduction in the number of measurements is directly related to the performance of the reconstruction algorithm employed in the post-processing phase, which is also a key element in compressive sensing systems. In this proposal we will also investigate the development of a novel reconstruction algorithm for PMMW imagery. However, due to the common properties in CS reconstruction and super-resolution, which is the second main objective in this proposal, a common algorithm can be developed that addresses both of these problems. This algorithm aims at reconstructing the original image from the compressive measurements and in parallel at improving the resolution of the final image. However, the algorithm can also be applied for each task independently as will be shown below. Before we describe the details of the proposed algorithm we provide a mathematical description of the PMMW acquisition system also accounting for degradations and noise.

Note that so far we have assumed that the PMMW system in (15) does not introduce any degradations in the acquired images. However, this is not realistic as the PMMW instrument introduces blurring and there exists a certain amount of acquisition noise present. These are also the problems addressed in PMMW super-resolution algorithms. A more realistic model therefore is given as

(16)

where is the system system matrix representing the system degradation

corresponding to the PSF , and represents the acquisition noise. In a single scan

PMMW system, is the identity matrix, and traditional PMMW systems aim at

reconstructing from . The compressive sensing system introduces another degradation

18

with , but the matrices and can be combined into a single system matrix. In the following, we will describe in detail the proposed algorithm which aims at reconstructing

from , which addresses both CS reconstruction and super-resolution issues within a general mathematical framework.

3.2.5.2 Proposed reconstruction/super-resolution method

Bayesian Modeling

In order to be able to reconstruct from the incoherent measurements , and the system

matrices and , both the observation process (15) and the unknown image have to be modeled. In this proposal we employ a Bayesian approach since the Bayesian framework provides a general treatment of all unknowns generalizing many existing algorithms, and it also provides state-of-the-art restoration performance.

We utilize a hierarchical Bayesian framework by employing a conditional distribution for the observation model in (15) and a prior distribution on the unknown image . These distributions depend on the model parameters β and α, which are called hyperparameters, and in the second stage of the hierarchical model we utilize additional prior distributions, called hyperpriors, to model them. This hierarchical Bayesian model utilized in carrying out a fully-Bayesian inference procedure leads to a fully-automated reconstruction algorithm.

In this proposal, we utilize the following factorization of the joint distribution of all unknown and observed quantities

(17)

In the following, we present the specific models utilized for each of these distributions.The observation noise is assumed to be independent and Gaussian with zero

mean and precision equal to β, that is, using (15),

(18)

The most important modeling choice is the modeling of the sparsity inherent in the target images. Improved reconstruction/super-resolution results are obtained with powerful image models. It is already well known that 2D images can be very accurately represented by only a few coefficients of a sparsifying transform, such as wavelet transforms or total-variation (TV) functions on the image. The PMMW images are expected to be mostly smooth except at a number of discontinuities (e.g., edges). As the spatial domain image priors, we will investigate the use of the total variation and wavelet functions because of their edge-preserving properties by not over-penalizing discontinuities in the image while imposing smoothness [Rudin:1992].

19

The original work on CS utilized regularization based on -norms [Candes:2006], [Donoho:2006]. In [Babacan:2009a], we have employed a signal prior based on hierarchical Laplace priors, which include -norm regularization as a special case. In

[Babacan:2009c], we developed a Bayesian algorithm based on non-convex -norms with 0<p<1, which enforce sparsity to a greater extent. These types of regularization functions are proven to be very well suited for CS reconstruction for a general class of signals. On the other hand, for imaging applications, priors of the form (18), and especially based on TV functions are shown to provide higher reconstruction performance [Candes:2006, Lustig:2007]. Based on our earlier work [Babacan:2008a] [Babacan:2008b] [Babacan:2008c], the TV functions result in non-linear and spatially-varying restorations and provide state-of-the-art reconstructions. Therefore, in this proposal we will concentrate on priors such priors, resulting in non-linear and spatially-varying recovery algorithms.

Specifically, we will consider image priors of the form

(19)

where

with and the horizontal and vertical difference operators at pixel j, respectively.

Finally, hyperpriors on the hyperparameters have to be formed. We will consider the use of Gamma priors as they are conjugate to the product of the distributions in (18) and (19), and they include the uniform priors as special cases.

Bayesian Inference

The Bayesian inference is based on the posterior distribution

(20)

However, the distribution is intractable, and therefore approximations are utilized. A common one is to approximate the posterior by a delta function at its mode. Then, the unknowns can be found by

(21)

Note that this formulation results in the well-known maximum a posteriori (MAP) estimates of . Specifically, assuming uniform hyperpriors on the hyperparameters, the

20

estimates found by this inference procedure are equivalent to the solution of the following regularized inverse problem:

(22)

The optimization problem in (21) can be solved using an alternating optimization scheme where the joint probability distribution is maximized for each unknown in an alternating fashion. We proceed next by developing the specific solutions for each of the unknown variables using (21). The estimate for the unknown image can be calculated as

(23)

(24)

The last terms in represent the smoothing terms in the reconstruction, where is a

diagonal matrix with elements . Clearly at the edges the entries of

become small thus attenuating the enforcement of the smoothing term and resulting in sharper reconstructed edges. This diagonal weighting results in a spatially-adaptive reconstruction which is known to achieve higher reconstruction performance than stationary algorithms. Additionally, we will investigate other image priors within this proposal, which also result in spatially-varying restoration algorithms. Examples include

-norm-based (Laplace) and -norm-based wavelet priors [Babacan:2009a], [Babacan:2009c].

Similarly to the image estimate, the hyperparameters can be found by solving (20) with respect to each hyperparameter and keeping the others constant. Proceeding in this fashion, the corresponding estimates can be found as [Babacan:2008c]

(25)

(26)

where a and b are parameters of the Gamma hyperpriors placed on . Finally, the algorithm iterates among estimating the unknown image using (23)

and the hyperparameters using (25) and (26) until a convergence criterion is met. It should be emphasized that this algorithm does not include any free parameters that should be tuned by the user specific to the application, which widens its applicability. Moreover, note that the optimization framework in (22) we considered here is based on MAP estimates of the unknowns. However, there are more powerful (but less computationally efficient) approaches that generally provide higher quality images. A

21

very attractive option is variational Bayesian analysis, which provides estimates of the probability distributions of the unknowns , rather than point estimates as in (23), (25), (26). These distribution estimates also account for estimation uncertainties, and therefore reduce the propagation of errors between estimations of unknowns. We have developed a number of restoration algorithms based on variational Bayesian analysis, which are currently among the state-of-the-art methods in image restoration, blind deconvolution and super-resolution [Babacan:2008b] [Babacan:2008c] [Babacan:2009d] [Molina:2008] [Molina:2006]. One of our primary goals in this proposal is to extend and adapt some of our variational Bayesian methods to the PMMW reconstruction/super-resolution problem.

Note that the proposed algorithm recovers the original image from degradations originating from the compressive measurement process and the PSF of the PMMW instrument. However, it can also be utilized for the sole purpose of super-resolution where the system matrix is set to identity. Therefore, the proposed method can additionally be seen as a novel method for PMMW super-resolution. Moreover, as mentioned in Section 3.2.4, existing methods for PMMW super-resolution are based on somewhat outdated methods, whereas the proposed method is based on state-of-the-art non-linear and spatially-varying approaches for image recovery with fully-automated Bayesian inference. Therefore, the proposed research is expected to result in the development of state-of-the-art methods for PMMW super-resolution as well as novel fast imaging systems for PMMW.

Experimental Results

Next we first provide some restoration examples from our earlier work presented in [Babacan:2009d] that demonstrate the effectiveness of the proposed non-linear and spatially varying restoration algorithm in Eq. (24). Our deconvolution algorithm based on the estimation procedure outlined above is applied to both a synthetic and a real image of Saturn, which was taken at the Calar Alto Observatory in Spain, shown respectively in Figs. 5(a) and (d). In Figs. 5(b) and 5(e) the restored images by [Molina:2006], another state-of-the-art method are shown. The algorithms in [Babacan:2008c] and [Babacan:2009d] are non-linear and spatially varying restoration method based on TV-image priors; the first one performs non-blind and the second one blind restoration. They were used to produce the restored images in Figs. 5(c) and 5(f), respectively. It is clear that our method removes the blur very effectively and provides a sharper image with fewer ringing artifacts than the method in [Molina:2006].

22

(a) (b) (c)

(d) (e) (f) Figure 5. Example restoration results: (a) Observed blurred noisy image, (b) Restoration result by [Molina:2006], (c) Restoration result by [Babacan:2008c]; (d) Observed Saturn image, (e) Restoration result by [Molina:2006], (f) Restoration result by [Babacan:2009d].

We now also provide some preliminary results based on our earlier work with novel compressive imaging applications, as this work is very similar in spirit to the system proposed in this work. In [Babacan:2009e], we proposed a novel camera design for the acquisition of light-fields, which are 4D small-baseline images of a scene. By exploiting the fact that different regions of a camera aperture correspond to images of the scene viewed from different angles, we incorporate a randomly coded mask in front of the aperture to obtain incoherent measurements of the incident light-field. These measurements are then decoded using a novel reconstruction algorithm to recover the original light-field image. We exploit the highly sparse nature of the light-field images to obtain accurate reconstructions with only a few measurements compared to the high angular dimension of the light-field image. Our design has many advantages: a) it is rather straightforward and inexpensive to implement, since it only consists of a consumer camera and an LCD as a spatial light modulator; b) the light field images are captured with much fewer acquisitions than traditionally needed; and c) the proposed design provides images with high spatial resolution and signal-to-noise-ratio (SNR), and therefore does not suffer from limitations common to existing light-field camera designs [Georgiev:2006].

23

Some results from [Babacan:2009e] are shown in Figs. 6 and 7. In Fig. 6, reconstruction errors are shown with varying number of measurements M. It is clear that very accurate reconstructions can be obtained using very few measurements. In fact, an average error of is already obtained with only 7 measurements. Examples of reconstructed images using 11 and 21 measurements are shown in Fig. 7(b) and Fig. 7(c), respectively. Note that the reconstructed images are nearly indistinguishable from the original image, which is shown in Fig. 7(a) (size of the images are N=250x125). It can be observed that using the proposed design the number of acquisitions can be significantly reduced (by a factor between 1/7 to 1/4). Furthermore, the reduction in the number of acquisitions is expected to be much higher with larger light-field images, due to the increased level of sparsity.

Figure 6. Number of measurement M vs relative reconstruction error (average over 50 runs)

(a)

(b) (c)Figure 7. Reconstruction examples from [Babacan:2009e]. (a) Original angular image, reconstructed images from (b) 11 measurements (relative reconstruction

24

error = ) and (c) 21 measurements (relative reconstruction error =

).

The proposed algorithm is expected to provide a significant reduction in acquisition times, or an equivalent increase in the image quality for a given acquisition time. As mentioned earlier, a Hadamard mask design provides an increase in SNR compared to traditional single scan systems. The proposed compressive sensing system in combination with the developed reconstruction algorithm is expected to provide an additional factor of 4 to 5 increase in SNR, or an equivalent reduction in imaging times. This expectation matches the universally accepted standard in compressive sensing, and it is also validated by our earlier work in CS. As shown above, our developed system in [Babacan:2009e] is very similar in spirit to the application we propose here, and reductions around factors of 7 are easily achieved for lightfield acquisition systems. Therefore, an overall improvement in SNR by a factor which is the product of the factor provided by the mask with the factor provided by compressive sensing is expected in the proposed design over traditional single scan PMMW systems.

Overall, in this work, we propose to develop a novel imaging system and novel post-processing methods for high-quality reconstruction and super-resolution of PMMW images. The proposed imaging system is based on compressive sensing principles, and it aims at reconstructing PMMW images from much fewer measurements than traditionally needed. Moreover, due to the mask-based design and nonlinear reconstruction/smoothing process, the resulting images will be of much higher quality than traditional PMMW imagery. Second, we will develop novel CS reconstruction/super-resolution algorithms based on the state-of-the-art image recovery principles. The developed algorithms are based on non-linear and spatially-varying processing, which provides high-quality restorations, and fully-automated Bayesian analysis which makes the algorithms easy to use and extends their applicability to a wide range of imaging situations.

3.3. Risk Assessment

The PIs are uniquely qualified to carry out the proposed research with extensive experience in image recovery, super-resolution, compressive sensing, pattern recognition, and detection and classification. The proposed research and development builds on previous successful results obtained by the PIs. Our key collaborators and contributors to the project at the Argonne National Laboratory (ANL) are Drs. Paul Raptis, Nachappa Gopalsami, and Alexander Heifetz. They have extensive experience on the design and implementation of PMMWI systems, as well as, on data analysis. ANL will provide the hardware implementation of the PWWI system, as well as, the data to be used in validating our approaches. Collectively, our team thus greatly increases the likelihood the proposed research is achievable in pursuit of the stated mission objective.

25

3.4. Research Duplication

The proposed work will build on the PIs’ previous research results and extend them in several ways. The research experience of the PI on the topics addressed in this proposal (image recovery, super-resolution, and compressive sensing) spans a 30 year period of continued contributions. The PI has been in the forefront of R&D in this area and is very knowledgeable of the developments in the field (one has to perform the due diligence every time a paper is submitted for publication). The post-doctoral student (Babacan) under consideration for the project is already a major contributor to the field, as can be judged from his list of publications. Both graduate students under consideration for the project (Ms. Wang and Mr. Amizic) already have a couple of publications on the topics addressed in this proposal. There is therefore zero probability in duplicating any of the research efforts in this area. The proposed work complements in a great way the research and development effort at ANL. Furthermore, it is expected that the signal processing algorithms that will result from this effort will be of use to other imaging projects within NNSA.

3.5. Management Plan and Budget taking into account Programmatic Balance & Value and Present & Past Performance

The project will be lead by Profs. Aggelos K. Katsaggelos and Thrasyvoulos N. Pappas and will employ one full-time post-doctoral student and two full-time graduate (Ph.D.) students. Our collaborators at Argonne National Laboratory are Dr. Paul Raptis, Systems Technology and Diagnostics Manager, Dr. Nachappa Gopalsami, PI on the PMMW sensors projects, and Dr. Alexander Heifetz who will be the liaison between Northwestern and ANL, all providing a tight link between the research activities at the two institutions. The PIs will closely supervise the students, set the project goals, monitor progress, and adjust research directions as needed based on intermediate results. The post-doctoral student is expected to have a coordinating role as well. The post-doctoral student and graduate students will be responsible for carrying out most of the research, as well as for the development and testing of the proposed techniques. For the graduate students this work will form the basis of their doctoral thesis. We believe that all requested personnel (one post-doc and two graduate students) are needed to carry out the project objectives. There are many aspects to the proposed work and there is a specific role in mind for each one of the team members. In addition, the post-doc student and graduate students considered for this project are highly qualified and very well-prepared to undertake the proposed tasks. Rapid progress is expected in advancing on the various topics proposed here and in completing the challenging tasks in the allotted time.

4. Relevance and Outcomes/Impacts The goal of the proposed research is to develop advanced image

processing algorithms which will allow the modification of existing single pixel PMMWI systems so that their usefulness is increased by considerably reducing acquisition time and improving their spatial resolution capabilities. PMMWI systems are very instrumental in

26

detecting thermal signatures, building structures, plumes, and traffic near facilities, over long distances and under adverse environmental conditions. They can be used for the search and monitoring of clandestine nuclear facilities, a central objective of the NA-22 Office. The outcome of this research effort will be portable, inexpensive, and very efficient and accurate PMMWI systems. It will greatly compliment the on-going work at the Argonne National Laboratory and it will also provide useful techniques that can potentially greatly benefit other imaging efforts at National Laboratories funded by DOE/NNSA/NA-22.

5. Project Timetable

Year 1

During the first year the focus of the project will be on the algorithmic developments. Separate algorithms for compressive sensing and super-resolution will be initially developed. Various masks will be implemented by the algorithms and various formulations of the Bayesian inference will be considered. For example, various prior models will be considered (e.g., TV and wavelets), as well as, point estimators (MAP estimation) and variational approximations of the posterior probability density functions. Synthetic data will be utilized as well as preliminary data acquired at ANL. The implementation of various masks will be considered by ANL during the first year.

Year 2

The collection of PMMW data will continue and be intensified during this year. Various masks, experimental set-ups, and experiments will be performed during this year. The data will be used by the various versions of the compressive reconstruction and super-resolution algorithms that were developed during the first year. This experimentation will lead to potential modifications to the algorithms as well as the experimental setup. As stated in the body of the proposal the number of acquisitions depends on the performance of the reconstruction algorithms. It is expected that random masks (elements with real values between zero and one drawn from rectified Gaussian or uniform distributions) will perform better (i.e., they will require fewer acquisitions) than the Hadamard mask at the expense of more complicated implementations.

Year 3

A convergence will be reached at the beginning of the 3rd year regarding the most efficient implementations of the system (use of masks, means to change its values, experimental set-ups, etc.) The system will be fully validated and its gains in terms of acquisition speed, increased resolution, and increased accuracy in plum detection will be established. Comparisons with other state-of-the-art systems (single pixel and focal arrays) will be performed, which might require the interaction and collaboration with

27

other facilities. The teaching of the previous two years will be utilized in possibly investigating additional directions, such as the super-resolution from multiple images of the same scene with sub-pixel shifts, or direction we have not envisioned at this stage of the proposal. Final reports will be prepared during this year.

6. Roles of Participants

The project will be directed by Prof. Katsaggelos. Prof. Pappas will have a co-supervisory role. The post-doctoral student to be involved is Derin Babacan who will be graduating with his PhD in December of 2009. He has extensive experience on the topics addressed in this proposal, as is evident from the list of references. He will be responsible for the coordination of the project and will provide his expertise towards the development of both the compressive sensing and super-resolution algorithms. He will be developing algorithms, based on his previous work, but will also work closely with the students. He will be responsible for the integration of the compressive sensing and super-resolution algorithms. The one graduate student, Jingnan Wang, is a 2nd year PhD student. She will be responsible for the compressive sensing work since it is in line with her previous research activities. The 2rd graduate student, Bruno Amizic, is also an experienced on the topic 2nd year student and he will be responsible for the super-resolution work and primarily for the data acquisition and the interface with ANL. It is expected that one of the graduate student will be spending a good amount of time at ANL, although the rest of the team will also have a close and frequent interaction with the ANL group. The team at ANL we will be primarily interacting with consists of Drs. Raptis, Gopalsami, and Heifetz (Dr. Heifetz will act as the liaison between ANL and Northwestern).

10. Statement of Objectives

Title of work to be performed Compressive Sensing and Super-Resolution of Passive Millimeter Wave Images

A. Objectives The objectives of the proposed research are to develop novel single pixel PMMWI systems and algorithms for the compressed sensing of images and their post-processing for increasing their resolution. These two tasks are tightly intertwined, as the specific interpretation of the acquired data through advanced signal processing techniques, allows for the design of sophisticated PMMWI structures of improved efficiency in terms of acquisition times and resolution of the imaged scene. The developed hardware and software will greatly advance the state of the art in PMMWI in creating inexpensive portable systems of high efficiency and accuracy. The advanced signal processing

28

techniques to be developed will also find applications with other imaging modalities. This represents a truly collaborative effort between Northwestern and the Argonne National Labs (ANL). Northwestern will provide primarily the algorithmic know-how while ANL will provide the required hardware implementation, the resulting data and will validate the results. Algorithms developed in this study will have a broad applicability to other imaging programs within NNSA.

B. Scope of the work

We propose to utilize hierarchical Bayesian approaches towards compressive sensing and super-resolution of PMMW images. Based on our previous work, such approaches have shown to exhibit a number of advantages in modeling the original scene and the noise, and in estimating all the unknown parameters. They result in non-linear and spatially adaptive recovery (both reconstruction and super-resolution) algorithms, capable of improving both the acquisition speed (via reduced acquisitions) and the spatial resolution of the reconstructed image. We propose to develop new algorithms building on our previous experience and expertise. An existing PMMWI system at ANL will utilized for acquiring compressed sampled iamges. Data under various configurations of the system and experimental setting will be collected and processed. Based on the results we will obtain we will modify both the algorithms and the system in order to improve its performance. In this manner, an outperforming system configuration (both hardware and software) will result. The overall effort will be undertaken by two faculty members, one post-doctoral student and two graduate students at Northwestern. A team of colleagues at ANL will closely collaborate with us in providing hardware implementations, data, and in evaluating the results.

C. Tasks to be performed

Task 1.0 Project management and Planning

Task 2. 0 Development of algorithms resulting from the inversion of the Hadamard maskSubtask 2.1 Investigation of regularized inverse solutions

Task 3.0 Development of algorithms for the reconstruction of compressively sampled PMMW imagesSubtask 3.1 Investigation of various coded masksSubtask 3.2 Investigation of various prior modelsSubtask 3.3 Investigation of various Bayesian inference/optimization techniquesSubtask 3.4 Use of synthetic data and preliminary real data

Task 4.0 Development of algorithms for super-resolution of PMMW imagesSubtask 4.1 Investigation of various prior modelsSubtask 4.2 Investigation of various Bayesian inference/optimization techniquesSubtask 4.3 Use of synthetic data and preliminary real data

29

Task 5.0 Development of algorithms combining reconstruction of compressively scanned and super-resolution of PMMW imagesSubtask 5.1 Use of synthetic data and preliminary real data

Task 6.0 Collection of PMMW images at ANLSubtask 6.1 Experimentation with various masksSubtask 6.2 Experimentation with various imaging scene

Task 7.0 Processing of PMWW imagesSubtask 7.1 Experimentation with all developed reconstruction algorithmsSubtask 7.2 Experimentation with all developed super-resolution algorithmsSubtask 7.3 Experimentation with all joint reconstruction and super-resolution algorithms

Task 8.0 Evaluation of the experimental resultsSubtask 8.1 Modification of the algorithmsSubtask 8.2 Modification of the experimental set-upSubtask 8.3 Determination of converged system

Task 9.0 Testing of resulting PMMW imaging systems and signal processing softwareSubtask 9.1 Performance evaluationSubtask 9.2 Sensitivity analysisSubtask 9.3 Comparison with other state-of-the art PMMWI systems

Tasks 2-5 will be primarily undertaken during the 1st year of the project, tasks 6-8 during the 2nd year and task 9 during the 3rd year of the project.

D. Deliverables

Periodic, topical, and final reports will be submitted in accordance with the “Federal Assistance Reporting Checklist” and the instructions accompanying the checklist.

30

APPENDIX A: Facilities and Other Resources

Facilities

The department of Electrical Engineering and Computer Science at Northwestern University is part of the Robert R. McCormick School of Engineering and Applied Science and consists of 50 faculty members including seven Chaired Professors, 12 faculty members who have received NSF PYI/NYI/CAREER awards, five McCormick Professors of Teaching Excellence, ten IEEE Fellows, two ACM Fellows, one APS Fellow, five SPIE Fellows, and two OSA Fellows. Two of our emeritus faculty members are in the National Academy of Engineering.

Every faculty member, postdoctoral researcher, and graduate student is provided with a PC with 100 Mbit full duplex Internet connections. Our building has a switched 100 Mbit and Gbit networking infrastructure, with 2 gigabit pipes to the rest of campus. The campus peers with multiple commodity providers as well as research networks such as Abilene and Starlight. We are a part of Microsoft's MSDNAA, which provides free or very inexpensive access to Microsoft products, which are used widely in desktop computing. Linux is also used on the desktop.

EECS laboratories and classrooms are located in the 750,000 square feet Technological Institute, which houses the McCormick School. The PIs' students have offices in the Tech Institute and the Ford Engineering Design Center, which is connected to the Technological Institute via a sky bridge. Our labs make use of over 1000 square feet of space in the Ford building, and we also have machine room space in the Technological Institute's machine room and in an open machine room run by Northwestern's Information Technology Group.

An infrastructure based on Unix (Linux and Solaris) and Windows AS servers is in place, and provides:

• DNS, DHCP, Active Directory and other basic network services. • Single sign-on supported by a combination of Windows and NIS systems. • Shared network file systems for home directories, available on both Windows and

Unix systems. • Network backup (VERITAS to a centralized tape robot). • Microsoft Exchange-based email, calendaring, discussion lists, and newsgroup

features, accessible using both MS and open protocols from all machines. • A dedicated staff of five systems support engineers is responsible for our

infrastructure and for assistance with research computing needs.

Other Resources

A dedicated eight member staff provides support for many areas, including purchasing, student advising and help, grant and proposal management, general financial management, and secretarial help.

31

APPENDIX B: Equipment Major Equipment The EECS Department has, among others, a large number of servers, as well as parallel and distributed infrastructure. These development servers (several of them) range from 128 node clusters to 8 node clusters with large memories per node and hundreds of terabytes of disk space. Furthermore, there are many specialized clusters such as FPGA clusters, NVIDIA GP/GPU clusters. In collaboration with the McCormick Engineering School, we have established a full featured machine room in the main engineering building. Currently, EECS has five racks of machines supporting various research projects in this space. We also have two smaller machine rooms in our building, which support our infrastructure machines, and several research projects.

Major Equipment Access

The EECS department has access to tera- and in the near future peta-scale systems through its participation with the Great Lake Consortium (led by NCSA) as well as its close collaboration with Argonne National labs. Particularly, we have access to generic clusters (several of them at NCSA), IBM Blue-Gene supercomputers. Several faculty involved in this project also have access to (via NSF and DOE allocations) supercomputers at Oak-Ridge National Labs (30K+ Cray XT4), San Diego Supercomputer Center systems, TACC (Texas) system, and several systems at the Lawrence Berkeley National Labs.

32

APPENDIX C: Bibliography and References

[Babacan:2009a] S.D. Babacan, R. Molina, and A.K. Katsaggelos, "Bayesian Compressive Sensing using Laplace Priors," to appear in IEEE Trans. on Image Processing, 2009

[Babacan:2009b] S. D. Babacan, L. Mancera, R. Molina, and A.K. Katsaggelos, "Bayesian Compressive Sensing Using Non-Convex Priors," Proc. European Signal Processing Conference 2009 (EUSIPCO'09), Glasgow, Scotland, Aug. 2009.

[Babacan:2009c] S. D. Babacan, R. Molina, and A.K. Katsaggelos, "Fast Bayesian Compressive Sensing using Laplace Priors," Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP’09), Taipei, Taiwan, April 2009.

[Babacan:2009d] S. D. Babacan, R. Molina, and A. K. Katsaggelos, "Variational Bayesian Blind Deconvolution Using a Total Variation Prior," IEEE Trans. on Image Processing, vol. 18, issue 1, pp. 12 - 26, Jan. 2009.

[Babacan:2009e] S. D. Babacan, R. Ansorge, M. Luessi, R. Molina, and A. K. Katsaggelos, "Compressive Lightfield Sensing," to appear, Proc. IEEE Int. Conf. on Image Processing 2009, Nov. 2009.

[Babacan:2008a] S.D. Babacan, R. Molina, and A.K. Katsaggelos, "Parameter Estimation in Total Variation Blind Deconvolution," Proc. European Signal Processing Conference 2008 (EUSIPCO'08), Lausanne, Switzerland, Aug. 2008.

[Babacan:2008b] S.D. Babacan, R. Molina, and A.K. Katsaggelos, "Total Variation Super Resolution Using a Variational Approach," Proc. IEEE Int. Conf. on Image Processing 2008, Oct. 2008.

[Babacan:2008c] S.D. Babacan, R. Molina, and A.K. Katsaggelos, "Parameter Estimation in TV Image Restoration Using Variational Distribution Approximation," IEEE Trans. on Image Processing, vol. 17, issue 3, pp. 326 - 339, March 2008.

[Banham:1997] M. R. Banham and A. K. Katsaggelos, “Digital Image Restoration,” IEEE Signal Proc. Magazine, Vol. 14, Issue: 2, pp. 24-41, March 1997.

[Bishop:2007] T. E. Bishop, S. D. Babacan, B. Amizic, A. K. Katsaggelos, T. Chan, and R. Molina, "Blind image deconvolution: problem formulation and existing approaches," Blind image deconvolution: theory and applications, P. Campisi and K. Egiazarian, editors: CRC press, 2007.

[Bolomey:2001] J.C. Bolomey and F.E. Gardiol, Engineering Applications of the Modulated Scatterer Technique, Artech House, 2001.

33

[Candes:2005] E. J. Candes and T. Tao, “Decoding by linear programming,” IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203–4215, 2005. [Candes:2006] E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inform. Theory, vol. 52, no. 2, pp. 489–509, Feb. 2006.

[Chantas:2009] G. Ghantas, N. P. Galatsanos, R. Molina, and A. K. Katsaggelos, “Variational Bayesian Image Restoration with a Product of Spatially Weighted Total Variation Image,” to appear, IEEE Trans. Image Processing.

[Donoho:2006] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inform. Theory, vol. 52, no. 4, pp. 1289–1306, 2006.

[Dai:2009a] W. Dai, M.A. Sheikh, O. Milenkovic and R. G. Baraniuk,. Compressive sensing DNA microarrays, EURASIP journal on bioinformatics & systems biology, 2009. [Dai:2009b] S. Dai, M. Han, W. Xu, Y. Wu, Y. Gong, , and A. K. Katsaggelos, "Soft Edge Smoothness Prior for Alpha Channel Super Resolution," IEEE Trans. Image Processing, vol. 18, issue 5, 969-981, May 2009.

[Decker:1970] J.A. Decker Jr., “Hadamard-Transform image scanning,” Appl. Opt. 9(6), 1392-1395, 1970. [Duarte:2008] M. Duarte, M. Davenport, D. Takhar, J. Laska, T. Sun, K. Kelly, and R. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Processing Magazine, 25(2), pp. 83 - 91, March 2008.

[Galatsanos:1992] N. P. Galatsanos and A. K. Katsaggelos, "Methods for Choosing the Regularization Parameter and Estimating the Noise Variance in Image Restoration and their Relation," IEEE Trans. Image Processing, vol. 1, 322-336, July 1992.

[Gan:2008] L. Gan, T. Do, and T. Tran, “Fast compressive imaging using scrambled block Hadamard ensemble,” Proc. EUSIPCO 2008, Lausanne, Switzerland, Aug. 2008. [Georgiev:2006] T. Georgiev, K. C. Zheng, B. Curless, D. Salesin, S. Nayar, and C. Intwala, “Spatio-angular resolution tradeoffs in integral photography,” Proc. EGSR, June 2006, pp. 263–272. [Goodman:1996] J. Goodman, Introduction to Fourier Optics, McGraw-Hill, New York, 1996.

[Gopalsami: 2008] N. Gopalsami, S. Bakhtiari, T. W. Elmer II, and A. C. Raptis, “Application of Millimeter-Wave Radiometry for Remote Chemical Detection,” IEEE Trans. On Microwave Theory and Techniques, vol. 56, no. 3, pp. 700-709, March 2008.

34

[Gopalsami:2009] N. Gopalsami, H. T. Chien, A. Heifetz, E. R. Koehl, and A. C. Raptis, “Millimeter wave detection of nuclear radiation: An alternative detection mechanism,” Review of Scientific Instruments, 80, 084702, 2009.

[Hunt:1992] B. R. Hunt, P. Sementilli, “Description of a Poisson Imagery Super Resolution Algorithm”, Astronomical Data Analysis Software and Systems 1, Ch. 23, pp. 196-199, 1992.

[Katsaggelos:1989] A. K. Katsaggelos, “Iterative Image Restoration Algorithms,” Optical Engineering, Vol. 28,   Issue 7, pp. 735-748, July 1989.

[Katsaggelos:1991a] A. K. Katsaggelos, editor , “Digital Image Restoration," Springer Series in Information Sciences, vol. 23: Springer-Verlag, 1991.

[Katsaggelos:1991b] A. K. Katsaggelos, J. Biemond, R. W. Schafer, and R. M. Mersereau, “A Regularized Iterative Image Restoration Algorithm,” IEEE Trans. On Signal Processing, Vol. 39, Issue: 4, pp. 914-929, April 1991.

[Katsaggelos:2007] A.K. Katsaggelos, R. Molina, and J. Mateos, "Super Resolution of Images and Video," Synthesis Lectures on Image, Video, & Multimedia Processing, Morgan & Claypool Publishers, 134 pages, 2007. 

[Katsaggelos:2009] A. K. Katsaggelos, S. D. Babacan, and C. Tsai, "Iterative Image Restoration," The Essential Guide to Image Processing, A. Bovik, editor, Elsevier, 2009.

[Lettington:1995] A. H. Lettington, Q. H. Hong, “Image restoration using a Lorentzian probability model”, Journal of Modern Optics, Vol. 42, No. 7, pp. 1367-1376, 1995. [Lettington: 2002] A. H. Lettington, M. R. Yallop, and D. Dunn, “Review of super-resolution techniques for passive millimeter-wave imaging,” in Proc. Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, R. Appleby, G. C. Holst, and D. A. Wikner, Eds., vol. 4719, Jul. 2002, pp. 230–239. [Lettington:2003] A. H. Lettington, D. Dunn, M. P. Rollason, N. E. Alexander, and M. R. Yallop, “Use of constraints in the super-resolution of passive millimeter-wave images,” in Proc. Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, R. Appleby, D. A. Wikner, R. Trebits, and J. L. Kurtz, Eds., vol. 5077, Aug. 2003, pp. 100–109. [Lucy:1974] L. B. Lucy, “An iterative technique for the rectification of observed distributions,” Astron. J., vol. 79, pp. 745+, June 1974. [Lustig:2007] M. Lustig, D. Donoho, and J. M. Pauly, “Sparse MRI: The application of compressed sensing for rapid MR imaging,” Magnetic Resonance in Medicine, 58(6) pp. 1182 - 1195, Dec. 2007.

35

[Marcia:2008] R. Marcia and R. Willett, ”Compressive coded aperture video reconstruction,” Proc. European Signal Processing Conf. (EUSIPCO), Lausanne, Switzerland, Aug. 2008.

[Marcia:2009] R. Marcia, Z. Harmany, and R. Willett, “Compressive Coded Aperture Imaging,” SPIE Electronic Imaging, 2009.

[Molina:2008] R. Molina, M. Vega, J. Mateos, and A.K. Katsaggelos, "Variational Posterior Distribution Approximation in Bayesian Super Resolution Reconstruction of Multispectral Images," Applied and Computational Harmonic Analysis, special issue on Mathematical Imaging, vol. 24, issue 2, 251-267, March 2008.

[Molina:2006] R. Molina, J. Mateos, and A. K. Katsaggelos, “Blind deconvolution using a variational approach to parameter, image, and blur estimation,” IEEE Trans. Image Processing, vol. 15, no. 12, pp. 3715–3727, Dec. 2006. [Pirogov:2003] Y. A. Pirogov, V. V. Gladun, I. V. Shlemin, S. P. Chzhen, D. A. Tischenko, A. L. Timanovskiy, A. V. Lebedev, “Super-resolution and coherent phenomena in multisensor systems of millimeter-wave radio imaging,” Proc. SPIE, Vol. 5077, pp. 110-120, 2003. [Pirogov:2004] Y. A. Pirogov, V. V. Gladun, D. A. Tischenko, A. L. Timanovskiy, I. V. Shlemin, S. F. Cheng, “Passive millimeter-wave imaging with super-resolution,” Proc. SPIE, pp. 72-83 vol 5573, 2004. [Richards:1988] M. A. Richards, “Iterative Noncoherent Angular Super-resolution”, Proc. of the IEEE National Radar Conference, pp. 100-105, 1988.

[Richardson:1972] H. W. Richardson, “Bayesian-based iterative method of image restoration,” Journal of the Optical Society of America, vol. 62, no. 1, pp. 55–59, Jan. 1972.

[Rudin:1992] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, pp. 259–268, 1992. [Schafer:1981] R. W. Schafer, R. M. Mersereau, and M. A. Richards, “Constrained Iterative Restoration Algorithms, Proc. IEEE, pp. 432-450, vol. 69, April 1981.

[SPM:2008] Compressive Sampling, Special issue, IEEE Signal Proc. Magazine, vol. 25, issue 2, March 2008.

[Sundareshan:2004] M. K. Sundareshan, S. Bhattacharjee, “Enhanced iterative processing algorithms for restoration and superresolution of tactical sensor imagery,” Optical Engineering, 43(1), PP. 199-208, Jan. 2004.

36

[Vega:2008] M. Vega, J. Mateos, R. Molina, and A. K. Katsaggelos, "Super-Resolution of Multispectral Images," The Computer Journal: Oxford University Press, 15 pages, June 2008.

[Vera:2009] E. Vera, L. Mancera, S.D. Babacan, R. Molina, and A.K. Katsaggelos, "Bayesian Compressive Sensing of Wavelet Coefficients Using Multiscale Laplacian Priors," Proc. IEEE Workshop on Statistical Signal Processing (SSP09), Cardiff, Wales, UK, Aug. 2009. [Xiao:1996] Z. Xiao, J. Xu, S. Peng, "Super resolution image restoration of a PMMW sensor based on POCS algorithm,” Proc. Systems and Control in Aerospace and Astronautics, ISSCAA 2006, 680-683J, 2006.

[Zheng:2007] X. Zheng and J. Yang , “Adaptive projected Landweber super-resolution algorithm for passive millimeter wave imaging,” Proc. SPIE 6787, 67871K (2007), DOI:10.1117/12.749949

[Zheng:2008a] X. Zheng, J. Yang, L. Li, Z. Jiang, “Wavelet-based super-resolution algorithms for passive millimeter wave imaging,” Proc. International Conference on Communications, Circuits and Systems, (ICCCAS 2008). pp. 826-829, May 2008.

[Zheng:2008b] X. Zheng, J. Yang, L. Li, Z. Jiang, “Modified Projected Landweber Super-resolution Algorithms for Passive Millimeter Wave Imaging,” Proc. Int. Conf. on Communications, Circuits and Systems, (ICCCAS 2008), pp. 789-792, May 2008.

37