breaking the coherence barrier - a new theory for ... · (ii)x-ray computed tomography...

Breaking the coherence barrier -A new theory for compressed sensing

Anders C. Hansen (Cambridge)

Joint work with:

B. Adcock (Simon Fraser University)C. Poon (Cambridge)

B. Roman (Cambridge)

iTwist, August 27, 2014

1 / 59

Compressed Sensing in Inverse Problems

Typical analog/infinite-dimensional inverse problem wherecompressed sensing is/can be used:

(i) Magnetic Resonance Imaging (MRI)(ii) X-ray Computed Tomography(iii) Thermoacoustic and Photoacoustic Tomography(iv) Single Photon Emission Computerized Tomography(v) Electrical Impedance Tomography(vi) Electron Microscopy(vii) Reflection seismology(viii) Radio interferometry(ix) Fluorescence Microscopy

2 / 59

Compressed Sensing in Inverse Problems

Most of these problems are modelled by the Fourier transform

F f (ω) =

∫Rd

f (x)e−2πiω·x dx ,

or the Radon transform Rf : S× R→ C (where S denotes the circle)

Rf (θ, p) =

∫〈x,θ〉=p

f (x) dm(x),

where dm denotes Lebesgue measure on the hyperplane x : 〈x , θ〉 = p.

I Fourier slice theorem ⇒ both problems can be viewed as theproblem of reconstructing f from pointwise samples of its Fouriertransform.

g = F f , f ∈ L2(Rd). (1)

3 / 59

Compressed Sensing

I Given the linear system

Ux0 = y .

I Solvemin ‖z‖1 subject to PΩUz = PΩy ,

where PΩ is a projection and Ω ⊂ 1, . . . ,N is subsampledwith |Ω| = m.

Ifm ≥ C · N · µ(U) · s · log(ε−1) · log (N) .

then P(z = x0) ≥ 1− ε, where

µ(U) = maxi ,j|Ui ,j |2

is referred to as the incoherence parameter.

4 / 59

Pillars of Compressed Sensing

I Sparsity

I Incoherence

I Uniform Random Subsampling

In addition: The Restricted Isometry Property + uniform recovery.

Problem: These concepts are absent in virtually all the problemslisted above. Moreover, uniform random subsampling gives highlysuboptimal results.

Compressed sensing is currently used with great success in many ofthese fields, however the current theory does not cover this.

5 / 59

Uniform Random Subsampling

U = UdftV−1dwt.

5% subsamp-map Reconstruction Enlarged

6 / 59

Sparsity

I The classical idea of sparsity in compressed sensing is thatthere are s important coefficients in the vector x0 that wewant to recover.

I The location of these coefficients is arbitrary.

7 / 59

Sparsity and the Flip Test

Let

x =

andy = Udfx , A = PΩUdfV

−1dw ,

where PΩ is a projection and Ω ⊂ 1, . . . ,N is subsampled with|Ω| = m. Solve

min ‖z‖1 subject to Az = PΩy .

8 / 59

Sparsity - The Flip Test

1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

x105

Truncated (max = 151.58)

Figure : Wavelet coefficients and subsampling reconstructions from 10% of Fourier coefficients with

distributions (1 + ω21 + ω2

2 )−1 and (1 + ω21 + ω2

2 )−3/2.

If sparsity is the right model we should be able to flip thecoefficients. Let

zf =

1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

x105

Truncated (max = 151.58)

9 / 59


I Lety = UdfV

−1dw zf

I Solvemin ‖z‖1 subject to Az = PΩy

to get zf .

I Flip the coefficients of zf back to get z , and let x = V−1dw z .

I If the ordering of the wavelet coefficients did not matter i.e.sparsity is the right model, then x should be close to x .

10 / 59

Sparsity- The Flip Test: Results

Figure : The reconstructions from the reversed coefficients.

Conclusion: The ordering of the coefficients did matter. Moreover, thisphenomenon happens with all wavelets, curvelets, contourlets andshearlets and any reasonable subsampling scheme.

Question: Is sparsity really the right model?

11 / 59


CS reconstr. CS reconstr, w/ flip Subsamplingcoeffs. pattern

512, 20%

UHadV−1dwt

FluorescenceMicroscopy

1024, 12%

UHadV−1dwt

CompressiveImaging,HadamardSpectroscopy

12 / 59

Sparsity - The Flip Test (contd.)


1024, 20%

UdftV−1dwt

MagneticResonanceImaging

512, 12%

UdftV−1dwt

Tomography,ElectronMicroscopy

13 / 59

Sparsity - The Flip Test (contd.)


1024, 10%

UdftV−1dwt

Radiointerferometry

14 / 59

Lesson Learned!

I The optimal sampling strategy depends on the sparsitystructure of the signal.

I Compressed sensing theorems must therefore show how theoptimal sampling strategy will depend on the structure.

I Such theorems currently do not exists.

15 / 59

What about the RIP?

I Did any of the matrices used in the examples satisfy the RIP?

16 / 59

Images are not sparse, they are asymptotically sparse

How to measure asymptotic sparsity: Suppose

f =∞∑j=1

βjϕj .

LetN =

⋃k∈NMk−1 + 1, . . . ,Mk,

where 0 = M0 < M1 < M2 < . . . and Mk−1 + 1, . . . ,Mk is the set ofindices corresponding to the kth scale.Let ε ∈ (0, 1] and let

sk := sk(ε) = minK :

∥∥∥ K∑i=1

βπ(i)ϕπ(i)

∥∥∥ ≥ ε∥∥∥ Mk∑i=Mk−1+1

βjϕj

∥∥∥,in order words, sk is the effective sparsity at the kth scale. Hereπ : 1, . . . ,Mk −Mk−1 → Mk−1 + 1, . . . ,Mk is a bijection such that|βπ(i)| ≥ |βπ(i+1)|.

17 / 59


0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

Relative threshold, ǫ

Sparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Level 6Level 7Level 8Worst sparsityBest sparsity

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


Sparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Level 6Level 7Level 8Worst sparsityBest sparsity

Figure : Relative sparsity of Daubechies 8 wavelet coefficients. 18 / 59


Curvelets

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

Relative threshold, ǫSparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Level 6Level 7Worst sparsityBest sparsity

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


Sparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Level 6Level 7Worst sparsityBest sparsity

Contourlets

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


Sparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Level 6Worst sparsityBest sparsity

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


Sparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Level 6Worst sparsityBest sparsity

Shearlets

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


Sparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Worst sparsityBest sparsity

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1


Sparsity,

s k(ǫ)/(M

k−

Mk−1)

Level 1Level 2Level 3Level 4Level 5Worst sparsityBest sparsity

19 / 59

Analog inverse problems are coherent

LetUn = UdfV

−1dw ∈ Cn×n

where Udf is the discrete Fourier transform and Vdw is the discretewavelet transform. Then

µ(Un) = 1

for all n and all Daubechies wavelets!

20 / 59

Analog inverse problems are coherent, why?

Note thatWOT-lim

n→∞UdfV

−1dw = U,

where

U =

〈ϕ1, ψ1〉〈ϕ2, ψ1〉 · · ·〈ϕ1, ψ2〉〈ϕ2, ψ2〉 · · ·

......

. . .

.Thus, we will always have

µ(UdfV−1dw ) ≥ c .

21 / 59

Analog inverse problems are asymptoticallyincoherent

Fourier to DB4 Fourier to Legendre Polynomials

Figure : Plots of the absolute values of the entries of the matrix U

22 / 59

Hadamard and wavelets are coherent

LetUn = HV−1

dw ∈ Cn×n

where H is a Hadamard matrix and Vdw is the discrete wavelettransform. Then

µ(Un) = 1

for all n and all Daubechies wavelets!

23 / 59

Hadamard and wavelets are asymptoticallyincoherent

Hadamard to Haar Hadamard to DB8 Enlarged

24 / 59

We need a new theory

I Such theory must incorporates asymptotic sparsity andasymptotic incoherence.

I It must explain the two intriguing phenomena observed inpractice:

I The optimal sampling strategy is signal structure dependentI The success of compressed sensing is resolution dependent

I The theory cannot be RIP based (at least not with theclassical definition of the RIP)

25 / 59

New Pillars of Compressed Sensing

I Asymptotic Sparsity

I Asymptotic Incoherence

I Multi-level Subsampling

26 / 59

Sparsity in levels

DefinitionFor r ∈ N let M = (M1, . . . ,Mr ) ∈ Nr with 1 ≤ M1 < . . . < Mr

and s = (s1, . . . , sr ) ∈ Nr , with sk ≤ Mk −Mk−1, k = 1, . . . , r ,where M0 = 0. We say that β ∈ l2(N) is (s,M)-sparse if, for eachk = 1, . . . , r ,

∆k := supp(β) ∩ Mk−1 + 1, . . . ,Mk,

satisfies |∆k | ≤ sk . We denote the set of (s,M)-sparse vectors byΣs,M.

27 / 59

Sparsity in levels

DefinitionLet f =

∑j∈N βjϕj ∈ H, where β = (βj)j∈N ∈ l1(N). Let

σs,M(f ) := minη∈Σs,M

‖β − η‖l1 . (2)

28 / 59

Multi-level sampling scheme

DefinitionLet r ∈ N, N = (N1, . . . ,Nr ) ∈ Nr with 1 ≤ N1 < . . . < Nr ,m = (m1, . . . ,mr ) ∈ Nr , with mk ≤ Nk − Nk−1, k = 1, . . . , r , andsuppose that

Ωk ⊆ Nk−1 + 1, . . . ,Nk, |Ωk | = mk , k = 1, . . . , r ,

are chosen uniformly at random, where N0 = 0. We refer to the set

Ω = ΩN,m := Ω1 ∪ . . . ∪ Ωr .

as an (N,m)-multilevel sampling scheme.

29 / 59

Local coherence

DefinitionLet U ∈ CN×N . If N = (N1, . . . ,Nr ) ∈ Nr and M = (M1, . . . ,Mr ) ∈ Nr

with 1 ≤ N1 < . . .Nr and 1 ≤ M1 < . . . < Mr we define the (k, l)th localcoherence of U with respect to N and M by

µN,M(k , l) =√µ(P

Nk−1

NkUP

Ml−1

Ml) · µ(P

Nk−1

NkU), k, l = 1, . . . , r ,

where N0 = M0 = 0.

30 / 59

The optimization problem

infη∈`1(N)

‖η‖`1 subject to ‖PΩUη − y‖ ≤ δ. (3)

31 / 59

Theoretical Results

Let U ∈ CN×N be an isometry and β ∈ CN . Suppose that Ω = ΩN,m is amultilevel sampling scheme, where N = (N1, . . . ,Nr ) ∈ Nr andm = (m1, . . . ,mr ) ∈ Nr . Let (s,M), where M = (M1, . . . ,Mr ) ∈ Nr ,M1 < . . . < Mr , and s = (s1, . . . , sr ) ∈ Nr , be any pair such that the followingholds: for ε > 0 and 1 ≤ k ≤ r ,

1 &Nk − Nk−1

mk· log(ε−1) ·

(r∑

l=1

µN,M(k, l) · sl

)· log (N) . (4)

Suppose that ξ ∈ CN is a minimizer of (3) with δ = δ√K−1 and

K = max1≤k≤r(Nk − Nk−1)/mk. Then, with probability exceeding 1− sε,where s = s1 + . . .+ sr , we have that

‖ξ − β‖ ≤ C ·(δ ·(1 + L ·

√s)

+ σs,M(f )),

for some constant C , where σs,M(f ) is as in (2), L = 1 +

√log2(6ε−1)

log2(4KM√s)

and

K = maxk=1,...,r

Nk−Nk−1

mk

.

32 / 59

Fourier to wavelets

Number of samples in each level:

mk & log(ε−1)· log(N) · Nk − Nk−1

Nk−1

·

(sk +

k−2∑l=1

sl · 2−α(k−l) +r∑

l=k+2

sl · 2−v(l−k)

),

where sk = maxsk−1, sk , sk+1. Note that

m & s · log(N), m = m1 + . . .+ mr , s = s1 + . . .+ sr .

Note that this shows how the success of CS will be resolutiondependent.

33 / 59

r-level Sampling Scheme

Figure : The typical sampling pattern that will be used.

34 / 59

Resolution Dependence, 5% subsampling

Size: 256× 256, Error = 10.8%

Original CS reconstruction Subsamp. map

35 / 59


Size: 512× 512, Error = 6.0%


36 / 59


Size: 1024× 1024, Error = 3.6%


37 / 59

2048× 2048 full sampling and 5% subsampling (DB4)

MRI Data courtesy of Andy Ellison, Boston University. Numericstaken from: On asymptotic structure in compressed sensing, B. Roman,

B. Adcock, A. C. Hansen, arXiv:1406.417838 / 59

The GLPU-Phantom

The Guerquin-Kern,Lejeune, Pruessmann, Unser-Phantom (ETHand EPFL)

39 / 59

Seeing further with compressed sensing

Figure : The figure shows 512× 512 full sampling (= 262144 samples)with 2048× 2048 zero padding.

Numerics taken from: On asymptotic structure in compressed sensing, B.

Roman, B. Adcock, A. C. Hansen, arXiv:1406.4178 40 / 59

Seeing further with compressed sensing

Figure : The figure shows 6.25% subsampling from 2048× 2048(= 262144 samples) and DB4.

Numerics taken from: On asymptotic structure in compressed sensing, B.

Roman, B. Adcock, A. C. Hansen, arXiv:1406.4178 41 / 59

Key Question

I Question: Will compressed sensing ever be used in commercialMRI scanners?

42 / 59

Siemens

Siemens has implemented our experiments and verified our theoryexperimentally on their scanners.

I See the Siemens report: ”Novel Sampling Strategies forSparse MR Image Reconstruction,” in Proceedings of theInternational Society for Magnetic Resonance in Medicine.

43 / 59

Siemens Conclusion:

I ”Significant differences in the spatial resolution can beobserved.”

I ”The image resolution has been greatly improved.”

I ”Current results practically demonstrated that it is possible tobreak the coherence barrier by increasing the spatial resolutionin MR acquisitions. This likewise implies that the full potentialof the compressed sensing is unleashed only if asymptoticsparsity and asymptotic incoherence is achieved. Therefore,compressed sensing might better be used to increase thespatial resolution rather than accelerating the data acquisitionin the context of non-dynamic 3D MR imaging.”

44 / 59

Key Question

I Question: Will Compressed sensing ever be used incommercial MRI scanners?

I Answer: We are closer than ever.

45 / 59

Electron Microscopy: Finte vs. infinite dimensions

Finite-dimensional setup:

minη∈CN

‖η‖l1 subject to PΩUdftV−1dwtη = PΩy .

Infinite-dimensional setup:

infη∈l1(N)

‖η‖l1 subject to PΩPNUη = PΩy ,

y = 〈f , ψ1〉, 〈f , ψ2〉, . . .

where

U =

u11 u12 u13 . . .u21 u22 u23 . . .u31 u32 u33 . . .

......

.... . .

, uij = 〈ϕj , ψi 〉.

46 / 59

Electron Microscopy: Finte vs. infinite dimensions

Original Original (zoom) Inf-dim. CS (zoom) Fin-dim. CS (zoom)Err 0.6% Err 12.7%

Figure : Subsampling 6.15%. Both reconstructions are based on identicalsampling information.

47 / 59

The infinite-dimensional CS model exists inengineering

The infinite-dimensional compressed sensing model has alreadybeen implemented by leading engineers, see

I M. Guerquin-Kern, M. Haberlin, K. Pruessmann, and M.Unser: ”A fast wavelet-based reconstruction method formagnetic resonance imaging”, IEEE Transactions on MedicalImaging, 2011.

48 / 59

Fundamental Question

I In a perfect world, would we want an MRI scanner that couldtake random Gaussian measurements?

49 / 59

Numerical example

Example: 12.5% measurements using DB4 wavelets.

256×256 512×512 1024×1024

Err = 41.6% Err = 25.3% Err = 11.6%

First case: Gaussian random measurements.

50 / 59

Numerical example

Example: 12.5% measurements using DB4 wavelets.

256×256 512×512 1024×1024

Err = 21.9% Err = 10.9% Err = 3.1%(41.6%) (25.3%) (11.6%)

Second case: Subsampled Fourier w/ multi-level sampling.

51 / 59

Fundamental Question

I In a perfect world, would we want an MRI scanner that couldtake random Gaussian measurements?

I Answer: NO!!!

52 / 59

Other work

Structured sparsity has been widely considered in CS.

I Tsaig & Donoho (2006), Eldar (2009), He & Carin (2009), Baraniuk

et al. (2010), Krzakala et al. (2011), Duarte & Eldar (2011), Som &

Schniter (2012), Renna et al. (2013), Chen et al. (2013) + others

Existing algorithms:

I Model-based CS, Baraniuk et al. (2010)

I Bayesian CS, Ji, Xue & Carin (2008), He & Carin (2009)

I Turbo AMP, Som & Schniter (2012)

Although the particulars are different, these are based on similarideas:

I Exploit the connected tree structure of wavelet coefficients

I Use standard measurements, e.g. random Gaussians

I Modify the recovery algorithm

53 / 59

Comparison with other structured CS algorithms

Multilevel subsampling with Fourier/Hadamard matrices

I Use standard recovery algorithm (l1 minimization)

I Exploit asymptotic sparsity in levels structure in the samplingprocess, e.g. Fourier/Hadamard

Other structured CS algorithms

I E.g. model-CS, Bayesian CS, Turbo AMP

I Use standard sensing matrices (random Gaussian/Bernoulli)

I Exploit connected tree structure by modifying the recoveryalgorithm

54 / 59

Comparison: 12.5% sampling at 256× 256 resolution

Original `1 Gauss., Err = 15.7% Model-CS, Err = 17.9%

BCS, Err = 12.1% TurboAMP, Err = 17.7% Mult. Four., Err = 8.8%55 / 59

Comparison: 12.5% sampling at 256× 256 resolution

Original `1 Bern., Err = 41.2% Model-CS, Err = 41.8%

BCS, Err = 29.6% TurboAMP, Err = 39.3% Mult. Four., Err = 18.2%56 / 59

Multlvl Fourier w/ other sparsifying transf.

wavelets, Err = 18.2% curvelets, Err = 17.4%

shearlets, Err = 16.5% TV, Err = 17.6%57 / 59

Final remarks

1. Standard CS using incoherent sensing matrices, e.g. randomGaussians, is highly suboptimal for imaging with -lets.

2. Images are not just sparse, but always possess a distinctasymptotic sparsity in levels structure.

3. Such structure can be exploited using multilevel subsamplingof Fourier/Hadamard matrices.

4. These matrices are not incoherent with wavelets, but have adistinct asymptotic incoherence structure.

5. By doing so, one obtains substantial improvements inaccuracy and computational efficiency over standard CS, andalso outperforms other structured CS algorithms.

58 / 59

Related Papers

For details see [4, 5, 2, 1, 3]

B. Adcock and A. C. Hansen.

Generalized sampling and infinite-dimensional compressed sensing.

Found. Comp. Math., 2014.

(in revision).

B. Adcock, A. C. Hansen, C. Poon, and B. Roman.

Breaking the coherence barrier: A new theory for compressed sensing.

arXiv:1302.0561, 2014.

A. C. Hansen.

On the solvability complexity index, the n-pseudospectrum andapproximations of spectra of operators.

J. Amer. Math. Soc., 24(1):81–124, 2011.

B. Roman, B. Adcock, and A. C. Hansen.

On asymptotic structure in compressed sensing.

arXiv:1406.4178, 2014.

Q. Wang, M. Zenge, H. E. Cetingul, E. Mueller, and M. S. Nadar.

Novel sampling strategies for sparse mr image reconstruction.

Proc. Int. Soc. Mag. Res. in Med., (22), 2014.59 / 59

breaking the coherence barrier - a new theory for ... · (ii)x-ray computed tomography...

Documents