compressive spectral image sensing, processing, and optimization

Compressive Spectral ImageSensing and Optimization

Gonzalo R. ArceCharles Black Evans Professor

University of DelawareNewark, Delaware, USA 19716

Distinguished Lecturer SeriesAristotle University of Thessaloniki

March 14, 2014

Optical imaging and spectroscopy discovers the characteristics of scenes andmaterials by capturing EM radiation in the 0.01 to 10000 nm spectrum window.

Sensitive not only to spatial and spectral information of a scene, but also topolarization, tomographic, angular, and even chemical composition.

Multidimensional imaging provides dimension preserving mappings

y = Hf,

where H is the “forward model“, characterizing the focal plane data.

Optical coding shapes H, under some criteria, to match the computational toolsof inverse problems to significantly improve the overall imaging performance.

MURA Hadamard SPECT

Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 2 / 50

THE SPECTRAL IMAGING PROBLEM?

Push broom spectral imaging: Traditional approach, expensive, lowsensing speed, senses N × N × L voxels

Optical Filters; Again senses N × N × L voxels; limited by number ofcolors


Compressive Spectral Imaging (CASSI), New revolutionary method, CSmakes a significant difference, senses only N2 N × N × L

Coded apertures arethe only variableelement.

Coded Apertures arethe key elements inCASSI.


WHY IS THIS IMPORTANT?Remote sensing and surveillance

Visible, NIR, SWIR

Devices are challenging in NIR and SWIR: cost, size,resolution, cooling


WHY IS THIS IMPORTANT?Medical Imaging: Vascular tissue imaging, angiography,contrast agent

paint restoration

Compressive Spectral ImagingReduce sensing complexity


Introduction

Compressive sensing introduced by Donoho†, Candès‡, Tao,Romberg...

Measurements are given by y = Φxy

xΦ

M x 1

Measurements M x N

Sampling Operator

N x 1

Sparse Signal

A sparse solution x is recovered from y by solving the inverseproblem

x = minx‖x‖1 s.t. y = Φx .

†Donoho. IEEE Trans. on Information Theory. December 2006.‡Candès, Romberg and Tao. IEEE Trans. on Information Theory. April 2006.


Introduction

Measurements are given by y = Φx

y ΨΦ α

x

A sparse solution α is recovered from y by solving the inverseproblem

α = minα‖α‖1 s.t. y = ΦΨα.


Introduction

Datacube

f = Ψθ

Compressive Measurements

g = HΨθ + w

Underdetermined system of equations

f = Ψminθ‖g− HΨθ‖2 + τ‖θ‖1


Coded-Aperture Spectral Imaging (CASSI)


Undetermined system of equations: N ×M × L Unknowns andN(M + L− 1) Equations.


Computational Model

A single shot compressive measurement across the FPA:

Gnm =L−1∑i=0

Fi(n+m)mTi(n+m) + win

F is the N ×M × L datacubeT is the binary code aperturew is the sensing noise

In vector form, the FPA measurement can be written as

g = Hf + w

H accounts for the aperture code and the dispersive element.


CASSI Multishot Matrix Model

g0

g1

...gk−1

=

H0

H1...

Hk−1

f, (1)

g = Hf, (2)

where H ∈ 0, 1N(M+L−1)K×NML.

Multi-shot coding done by using multiple coded apertures or aDigital-Micromirror-Device (DMD)


Matrix CASSI representation g = Hf

Data cube:N × N × L

Spectral bands: L

Spatial resolution:N × N

Sensor sizeN × (N + L− 1)

V=N(N+L-1)


Coded Aperture Optimization: Sensing andReconstruction

Restricted Isometry Property in CASSIGiven

g = HΨ︸︷︷︸A

θ with |θθθ| = S

The RIP of the CASSI matrix A is defined as the smallest constant δssuch that

(1− δs) ||θθθ||22 ≤ ||Aθθθ||22 ≤ (1 + δs) ||θθθ||22, (3)

whereδs = max

T⊂[n],|T |≤S

√λmax

(A|T ||T | − I

)(4)

A|T ||T | = AT|T |A|T |, A|T | is a m × |T | matrix whose columns are equal to

|T | columns of the CASSI matrix A, and λmax (.) denotes the largesteigenvalue.


Let the entries of ΨΨΨ be Ψj,k and let the columns of ΨΨΨ be [ψψψ0, . . . ,ψψψn−1].The entries of A|T | can be written as(

A|T |)

jk = (HψψψΩk )j = hTj ψψψΩk

=L−1∑r=0

(ti)

j−rNΨj+r(N′),Ωk

for j = 0, . . . ,m − 1, k = 0, . . . , |T | − 1, where i = bj/Vc, N ′ = N2 − N,and Ωk ∈ 0, . . . ,n − 1. The entries of A|T ||T | can be expressed as

(A|T ||T |

)jk =

K−1∑i=0

V−1∑`=0

L−1∑r=0

L−1∑u=0

(ti)`−rN

(ti)`−uN

Ψ`+rN′,Ωj Ψ`+uN′,Ωk (5)

for j , k = 0, . . . , |T | − 1.

Note that the coded aperture products(ti)

`−rN

(ti)

`−uN determine theeigenvalues of A|T ||T |, and consequently they determine the constantδs in the RIP.


Boolean Coded Apertures

An optimal ensemble of four 64× 64 boolean coded apertures.

Each spatialcoordinate in theensemble containsonly one 1-valueentry.


Bernoulli Coded Apertures

An ensemble of four 64× 64 Bernoulli coded apertures.


Performance of coded apertures


Original datacube Boolean (40.4dB) Unsigned grayscale (31.2dB)

Binary (27.7dB) Hadamard (27.7dB) Signed grayscale (22.7dB)


Original datacube Boolean Unsigned grayscale

Binary Hadamard Signed grayscale


Coded Aperture Optimization: Spectral Selectivity

UAV sensor requirements depend on flight duration, range,altitude, etc.Need: Hyperspectral imaging that dynamically adapts to optimalspectral bands.


((a)) Original ((b)) 12 Random Codes ((c)) 9 Optimal Codes

The resulting spectral data cubes are shown as they would be viewed by a StingrayF-033C CCD Color Camera.


((a)) Random Code ((b)) Original ((c)) Optimal Code

((d)) Random Codes ((e)) Optimized Codes


((a)) Random ((b)) Optimized

((c)) ((d))

Differences between the original and the reconstructed 3rd spectral channel (479nm)


((a)) Original ((b)) 12 Random Codes ((c)) 12 Optimized Codes

The resulting spectral data cubes are shown as they would be viewed by a StingrayF-033C CCD Color Camera.


Coded Aperture Optimization: Image Classification

Goal: classification of a spectral scene usingCompressive measurementsOptimal code apertureSparsity signal model


Given the FPA measurements g, every test pixel fi belongs to one ofthe P known classes

H(1),H(2), ...,H(P)

0 10 20 30 40 50 60 700

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Spectrum Band

Glutamine

0 10 20 30 40 50 60 700

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Spectrum Band

Histidine

0 10 20 30 40 50 60 700

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Spectrum Band

Isoleucine

If a pixel fi ∈ H(k), its spectral profile lies in a low-dimensionalsubspace spanned by the training samples: s(k)

j j=1,...,Np

fi ≈ [s(k)1 ,s(k)

2 , ...,s(k)Nk

][α(k)1 , α

(k)2 , ..., α

(k)Np

]T = S(k)α(k)

where, α(k) is a sparse vector.Gonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 30 / 50

Sparsity Model

Combining all class sub-dictionaries

fi ≈ [S(1), ...,S(k), ...,S(P)][α(1), ...,α(k), ...,α(P)]T = Sα

Ideally, if fi ∈ H(k), thenα(j) = 0; ∀j = 1, ...,P; j 6= k .


Superposition of 3 shots for the different codes

Optimal codes

5 10 15 20 25 30

5

10

15

20

25

30

0

0.5

1

1.5

2

2.5

3

Hadamard Codes

5 10 15 20 25 30

5

10

15

20

25

30

0

0.5

1

1.5

2

2.5

3

Bernoulli codes

5 10 15 20 25 30

5

10

15

20

25

30

0

0.5

1

1.5

2

2.5

3

Ck =∑K−1

k=0 (tkm,n)2 at each (m,n) spatial location.

Optimal Hadamard BernoulliGonzalo R. Arce () Compressive Spectral Image Sensing Oct., 2013 32 / 50

Image Classification Results

200 spectral bands. 50 shots.Ground-truth Single spectral band SVM-full datacube (73.5%)

Bernoulli-25%datacube(64.3%)

Hadamard-25%datacube(63.4%)

Optimal-25%datacube (73.7%)


Image Classification Results

200 spectral bands. 50 shots.Ground-truth FPA measurement SVM-full datacube (73.5%)

Bernoulli-25%datacube(64.3%)

Hadamard-25%datacube(63.4%)

Optimal-25%datacube (73.7%)


Colored Coded Aperture Spectral ImagingPatterned coating combines micro-lithography with optical coatingtechnology.

Precision patterned coatingand patternsSub-pixel alignment accuracyUltraviolet, visible, NIR, SWIRMulti-filter arrays on monolithicsubstrates


NEW FAMILY OF CODED APERTURES

Boolean Spectrally Selective

Super-resolution Colored


Colored coded aperture model

Colored coded aperture is a color filter arrayEach entry is a wavelength selective color filter3D Mask model has the same dimensions than the objective discrete data cube


Linear dispersion and focal plane array integrationLinear shifting operation

Focal plane array (FPA) projections

The number of pixels of the FPAdetector is N(N + L− 1) N2L(size of the spectral data cube)


Random Boolean Code


4 Colors Random Code


Restricted Isometry Property of Colored CASSIA = HΨΨΨ, fff = ΨΨΨθθθ, ΨΨΨ = W⊗ΨΨΨ2D

Definition

(1− δs) ||θθθ||22 ≤ ||Aθθθ||22 ≤ (1 + δs) ||θθθ||22,

δs = maxT ⊂[N2L],|T |≤S

||AT|T |A|T | − I||22,

A|T |, |T | columns of A indexed by the set T

δs = maxT ⊂[N2L],|T |≤S

λmax(A|T ||T | − I

)(A|T |

)ir

= hiψψψΩr

=

L−1∑k=0

(tìk

)mi−kN

Ψmi +k(N′),Ωr

(hi )j =

(

tìkj

)i−ì V−kj N

, if i − ì V = j − kj N′

0, otherwise,

A = HΨΨΨ2D

A = H(W⊗ΨΨΨ2D)


A|T ||T | = AT|T |A|T |

(A|T ||T |

)r ,u =

K−1∑`=0

V−1∑i=0

L−1∑k1=0

L−1∑k2=0

(t`k1

)i−k1N

(t`k2

)i−k2N

Ψi+k1N′,ΩrΨi+k2N′,Ωu


Results: LH-Colored Coded Aperture

A geometric interpretation of the colored coded apertures for LH-Colored filters (3shots).


Code Design Results B-Colored Coded Apertures (Band Pass Filters)

Geometric interpretation of colored coded apertures for B-filters (3 shots)


461nm 470nm 479nm 488nm

497nm 506nm 515nm 524nm

533nm 542nm 551nm 560nm

569nm 578nm 587nm 596nm

16 channels461-596nm256× 256pixels


Reconstruction From B-Colored Coded Apertures

2 4 6 8 10 12 14

30

40

50

60

70

80

90

Number of Shots

PSNR

Block Unblock

Random B-colored

B-colored

Mean PSNR of the reconstructed data cubes.


Reconstructions from Real Measurements (RGB)

K = 1 K = 1

Sh=1

(i) 1 snapshot: Photomask, Coloring, Optimal

K = 4 K = 4

Sh=4

(j) 4 snapshots: Photoask, Coloring, Optimal


Reconstruction Results (Photomask vs ColoredCoded Apertures)

454 nm K = 1 468 nm K = 1 485 nm K = 1 506 nm K = 1 529 nm K = 1 557 nm K = 1 594 nm K = 1 639 nm K = 1


(a) One snapshot reconstruction. (First row) Photomask, (Second row) Coloring



(b) 4 snapshots reconstructions. (First row) Photomask, (Second row) Coloring


Spectral Reconstruction From Colored Coded Apertures

Sh=4

P1

P2

454 468 485 506 529 557 594 6390

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Nanometers

Norm

. In

tensity

Spectral Comparison: 4 Snapshots

Spectrometer

Photomask

Coloring

Optimal

454 468 485 506 529 557 594 6390

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Nanometers

Norm

. In

tensity

Spectral Comparison: 4 Snapshots

Spectrometer

Photomask

Coloring

Optimal

Pixel P1

Pixel P2


Conclusions

Optical Coding for Compressive Spectral ImagingGood codes for reconstruction, classification, unmixingColored codes offer multidimensional coding - open problem

Convolution Optical Coding (Projections)Light field imagingX-ray tomographySuper-resolution microscopy


compressive spectral image sensing, processing, and optimization

Technology