model-based image reconstruction of chemiluminescence...

Model-based image reconstruction ofchemiluminescence using a plenoptic 2.0 camera

Hung Nien, Jeffrey A. Fessler, and Volker Sick

EECS Department and Mechanical Engineering DepartmentUniversity of Michigan, Ann Arbor

ICIP 2015: Computational ImagingSept. 29, 2015

1 / 29

Disclosure

I Supported by NSF under grant number CBET-1402707

I Equipment support from Intel Corporation

2 / 29

Motivation: combustion in transparent engine cylinder

3 / 29

Motivation

Tomographic reconstruction of 3D chemiluminescence patternssuch as flame fronts using a plenoptic camera.

Previous workI Tomo-PIV (particle image velocimetry) (4–6 cameras) [Elsinga et al.,

2006]

I Plenoptic 1.0 camera for PIV [Fahringer et al., 2012]

I Single-camera stereo [Greene et al., 2013] [Chen et al., 2015]

Depth maps for translucent objects?

4 / 29

Plenoptic camera

Plenoptic cameras use micro-lens arrays to capture 4-D light fieldinformation of a scene. The angular information enables:

I depth estimation (for object surfaces illuminated externally)e.g., via triangulation [Perwaß, SPIE, 2012]

I tomographic reconstruction (for luminescent objects)(cf., digital X-ray tomosynthesis - limited-angle tomography).

� Images courtesy of Raytrix GmbH and Lytro, Inc.

5 / 29

Model-based image reconstruction (MBIR)

Overall goal: reconstruct 3D chemiluminescence pattern x fromplenoptic camera measurement y.

MBIR components:I 3D object model (basis coefficients) x

- Image voxel, basis function, ...

I System model A (# of sensor elements × # of object voxels)

- Linearity, finite voxel size, finite pixel size, ...

I Data noise statistics p(y|Ax)

- Additive Gaussian, Poisson, ...

I Cost function Ψ(x)

- Data fidelity, regularizer, physical constraints, ...

I Iterative algorithm (arg minx)

- MART, FISTA, Newton’s methods, ...

[Nuyts et al., Phys. Med. Biol., 2013]

6 / 29

Model-based image reconstruction (continued)

We reconstructed objects by solving a regularized LS problem:

x? ∈ arg minx

{Ψ(x) , 1

2 ‖y − Ax‖22 + R(x)

}s.t. x � 0 ,

where R denotes an edge-preserving corner-rounded TV regularizer.

We focused on R defined as

R(x) ,∑

i=1 βi∑

n ϕHuber([Cix]n) ,

I Ci : finite difference matrix along ith direction

I βi : corresponding regularization parameter.

I ϕHuber(t) ≈ |t|

7 / 29

System model for a plenoptic camera

[Bishop & Favaro, IEEE T-PAMI, 2012]

8 / 29


Build (pre-compute) system matrix A one column at a time

[Bishop

& Favaro, IEEE T-PAMI, 2012]

8 / 29


| ←− z −→ | ←− Z −→ |thin lens formula: 1/z + 1/Z = 1/F

[Bishop & Favaro, IEEE T-PAMI,

2012]

8 / 29


[Bishop & Favaro, IEEE T-PAMI, 2012]

8 / 29


Use superposition to consider one microlens at a time

[Bishop &

Favaro, IEEE T-PAMI, 2012]

8 / 29


If microlens had large diameter...


2012]

8 / 29


If main lens had large diameter...


2012]

8 / 29


Combined effect of main lens and microlens

[Bishop & Favaro, IEEE

T-PAMI, 2012]

8 / 29

System model - continued

Continuous-space PSF of the ith micro-lens is:

βi (s, t; x , y , z) = βML-µLi (s, t; x , y , z)︸︷︷︸∝ circ

(s,t;cML-µL

i ,Bi

) ·βµLi (s, t; x , y , z)︸︷︷︸∝ circ

(s,t;cµL

i ,bi

) ,where

I (s, t) denotes 2D sensorcoordinates

I centers cML-µLi , cµL

i , and radii Bi ,

and bi depend on the object

point position (x , y , z) and

camera geometry.

I∑

i βi (s, t; x , y , z) sketched:

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

s−axis [mm]

t−axis

[m

m]

Continuous−space PSF

9 / 29

Computational challenges

I Dense micro-lens array

I Highly shift-variant point spread function

I Non-separable aperture / PSF (cf., X-ray CT)

I Lens aberrations

I Finite sensor pixel sizeThe discrete PSF of a micro-lens consists of integrals of the circle-circleintersection over each sensor pixel, where the circle centers depend on theposition of the “point source.”

We approximate each finite-sized sensor pixel as L× L infinitesimal pixels,

i.e., L×-subsampling in each direction.

I Finite object voxel size

10 / 29

Finite-sized object voxel effects

One (x , y) transaxial plane of a 3D object voxel

PSF

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

9.8x 10

−5 PSF zoom

s [mm]t

[mm

]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

9.8x 10

−5

11 / 29

Finite-sized object voxel: baseline approximation

We approximate each cubic voxel as K × K × K equally spacedinfinitesimal voxels, i.e., K×-subsampling in each direction.

PSF

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

4.8x 10

−3 PSF zoom

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

4.8x 10

−3

† We used K = 7 here.

12 / 29

Finite-sized object voxel: baseline approximation

We approximate each cubic voxel as K × K × K equally spacedinfinitesimal voxels, i.e., K×-subsampling in each direction.

PSF

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

9.8x 10

−5 PSF zoom

s [mm]t

[mm

]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

9.8x 10

−5

(infinitesimal)

12 / 29

Why finite voxel size matters

(zoomed)

Infinitesimal voxels

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

0.285

Finite−sized voxels

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

0.285

† Using 0.5× 0.5× 0.5 [mm3] cubic voxels and K = 7.

13 / 29

Why finite voxel size matters (zoomed)

Infinitesimal voxels

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

0.285

Finite−sized voxels

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

0.285

Abs. difference [0 0.034]

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

0.035

† Using 0.5× 0.5× 0.5 [mm3] cubic voxels and K = 7.

13 / 29

Numerical experiments: imaging geometry

14 / 29

Numerical experiments: object geometry

I 100× 100× 100 voxel object

I 0.5× 0.5× 0.5 [mm3] voxels

I 50 [mm] field-of-view

I 7× sensor subsampling whenprecomputing A

I 50 dB SNR (additive whiteGaussian noise)

I To avoid an inverse crime when synthesizing plenoptic sensor pictures, weused a voxelized object having a 2× finer grid in 3D, with 11×subsampling per dimension.

15 / 29

Numerical experiments: plenoptic cameras

Camera #1 Camera #2 Camera #3

fmain 80 80 80 [mm]f-number 1.4 2.8 1.4dmain 57.14 28.57 57.14 [mm]

fmicro 0.35 0.35 0.35 [mm]dmicro 0.27 0.135 0.135 [mm]

type larger Bishop & Favaro overlaps

I 9µm × 9µm sensor pixel size

I 850× 850 pixel sensor

16 / 29

Numerical experiments: simulated plenoptic pictures

Camera #1

s [mm]

-3 -2 -1 0 1 2 3

t [m

m]

-3

-2

-1

0

1

2

3

0

2.48

Better angular resolution than Camera #2, but worse spatial resolution

17 / 29


Camera #2

s [mm]

-3 -2 -1 0 1 2 3

t [m

m]

-3

-2

-1

0

1

2

3

0

2.47

Bishop & Favaro, 2012

17 / 29


Camera #3

s [mm]

-3 -2 -1 0 1 2 3

t [m

m]

-3

-2

-1

0

1

2

3

0

1.37

Overlapping (larger) subimages: demultiplexing needed,

but (perhaps) more angular information than Camera #2.

17 / 29

Numerical experiments: image reconstruction

[Recap] Our image reconstruction problem is:

x? ∈ arg minx

{Ψ(x) , 1

2 ‖y − Ax‖22 + R(x)

}s.t. x � 0 ,

I 500 iterations of FISTA with adaptive restart[Beck & Teboulle, IEEE T-IP, 2009]

[O’Donoghue & Candes, FCM, 2015]

I Precomputed / stored A (column-wise sparse)

- 7× object subsampling (x , y , z)- 7× sensor subsampling (s, t)- 320 secs/slice for camera #1

(60 threads @ MATLAB)

I 26 3D neighbors used in the (smoothed) TV regularizer

18 / 29

Numerical experiments: finite voxel size

Infinitesimal voxels Finite-sized voxels

Contours at isovalue = 20% of maximum intensity.

19 / 29

Numerical experiments: aperture size

Large aperture/coarse MLA Small aperture/dense MLA

Better lateral resolution of Camera 2 not helpful here.

20 / 29

Numerical experiments: aperture size - slices

Phantom

xy yz xz

0

10

Camera #1 [approx., SNR = 50 dB]

xy yz xz

0

3Camera #2 [approx., SNR = 50 dB]

xy yz xz

0

3

21 / 29

Numerical experiments: overlapping subimages

Non-overlapping subimages Overlapping subimages

22 / 29

Numerical experiments: overlapping subimages - slices

Phantom

xy yz xz

0

10


xy yz xz

0


xy yz xz

0

3

23 / 29

Numerical experiments: object distance

dobject = 700 [mm] dobject = 550 [mm]

24 / 29

Numerical experiments: object distance - slices

Phantom

xy yz xz

0

10


xy yz xz

0


xy yz xz

0

3

dobject = 700 [mm] dobject = 550 [mm]

25 / 29

Numerical experiments: sharp vs smooth object

Sharp object edges Smooth object edges

26 / 29

Numerical experiments: sharp vs smooth object - slices

Sharp phantom

xy yz xz

0

10Smooth phantom

xy yz xz

0

10


xy yz xz

0


xy yz xz

0

3

27 / 29

Conclusions

I Model-based image reconstruction may be viable for 3Dchemiluminescence from plenoptic camera data

I Voxel-size modeling is important

I Larger angular range of incident light improves z-resolution(but more severe lens aberration?)

I F-number matching can be relaxed (overlapping sub-images)to improve depth resolution in tomographic formulation

I Need fast on-the-fly forward/back-projections to solve reallarge-scale image reconstruction problems

28 / 29

Bibliography

[1] G. E. Elsinga, B. Wieneke, F. Scarano, and A. Schroder, “Tomographic 3D-PIV and Applications,” inParticle Image Velocimetry, Springer Berlin Heidelberg, 2008, pp. 103–25.

[2] T. W. Fahringer and B. S. Thurow, “Tomographic reconstruction of a 3-D flow field using a plenopticcamera,” in AIAA Fluid Dynamics Conf., 2012, p. 2826.

[3] M. L. Greene and V. Sick, “Volume-resolved flame chemiluminescence and laser-induced fluorescenceimaging,” Appl Phys B, vol. 113, no. 1, 87–92, Oct. 2013.

[4] H. Chen, P. Lillo, and V. Sick, “3D spray-flow interaction in a spark-ignition direct-injection engine,” Intl.J. Engine Research, 2015, To appear.

[5] C. Perwass and L. Wietzke, “Single lens 3D-camera with extended depth-of-field,” in Proc. SPIE 8291Human Vision and Electronic Imaging XVII, 2012, p. 829 108.

[6] J. Nuyts, B. De Man, J. A. Fessler, W. Zbijewski, and F. J. Beekman, “Modelling the physics in iterativereconstruction for transmission computed tomography,” Phys. Med. Biol., vol. 58, no. 12, R63–96, Jun.2013.

[7] T. E. Bishop and P. Favaro, “The light field camera: Extended depth of field, aliasing, andsuperresolution,” IEEE Trans. Patt. Anal. Mach. Int., vol. 34, no. 5, 972–86, May 2012.

[8] A. Beck and M. Teboulle, “Fast gradient-based algorithms for constrained total variation image denoisingand deblurring problems,” IEEE Trans. Im. Proc., vol. 18, no. 11, 2419–34, Nov. 2009.

[9] B. O’Donoghue and E. Candes, “Adaptive restart for accelerated gradient schemes,” Found.Computational Math., vol. 15, no. 3, 715–32, Jun. 2015.

29 / 29

Approximate box blur

In practice, Bi � bi , so βi is usually a circle.The continuous-space PSF of a voxel slice at depth z is

∫ x+4x

2

x−4x2

∫ y+4y

2

y−4y

2

βi (s, t; x , y , z) dydx

≈∫ x+

4x2

x−4x2

∫ y+4y

2

y−4y

2

circ(s, t; cµL

i (x , y) , bi)dydx

=

∫ 4x2

−4x2

∫ 4y

2

−4y

2

circ(s, t; cµL

i (x + δx , y + δy ) , bi)dδydδx

=

∫∫circ(s − δs , t − δt ; cµL

i (x , y) , bi)· rect(δs , δt ;ws ,wt) dδtdδs .

Back

29 / 29

Infinitesimal vs finite-sized sensor pixel: point objectInfinitesimal voxel/pixel

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

4.8x 10

−3 Infinitesimal voxel

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

4.8x 10

−3

Infinitesimal voxel/pixel

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

4.8x 10

−3 Infinitesimal voxel

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

4.8x 10

−3

(For an infinitesimal voxel)29 / 29

Infinitesimal vs finite-sized sensor pixel: sphere object


s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

0.285

Infinitesimal voxel

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

0.285

Finite−sized voxel/pixel

s [mm]

t [m

m]

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

0

0.285


s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

0.285

Infinitesimal voxel

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

0.285

Finite−sized voxel/pixel

s [mm]

t [m

m]

−0.2 −0.1 0 0.1 0.2

−0.2

−0.1

0

0.1

0.2

0

0.285

point voxel point voxel finite voxelpoint sensor finite sensor finite sensor

29 / 29

model-based image reconstruction of chemiluminescence...

Documents