JRF -1

Phaseless Imaging Algorithmsand Applications

James R. Fienup

Robert E. Hopkins Professor of OpticsUniversity of Rochester

Institute of Opticsfienup@optics.rochester.edu

Presented to:Institute for Mathematics and Applications

University of Minnesota

August 2017

JRF -2

•  Introductory phase retrieval example–  Astronomical imaging

•  Mathematical background–  The problem–  Iterative transform algorithms–  Nonlinear optimization algorithms

•  Recent application examples•  Lensless coherent optical imaging•  Non-crystallographic (x-ray) diffraction

Outline

JRF -3

Passive Imaging of Astronomical/Space Objects

blurredimage

object

aberrated optical system

}

LickObservatoryNaval Prototype Optical Interferometer

JWST

•  Solutions:–  Hubble Space Telescope (2.4 m diameter), $2 B; JWST $8 B–  Adaptive optics + laser guide star, $10’s M–  Optical interferometry, $10’s M–  Stellar speckle interferometry, << $1 M

•  Problem: atmospheric turbulence causes phase errors, limits resolution–  ≈ 1 arc-sec ≈ 5*10–6 rad. ≈ λ/ ro for λ = 0.5 μm and ro = 10 cm

•  Best resolution as though through only 10 cm aperture (vs. Keck 10 m)

JRF -4

Labeyrie’s Stellar Speckle Interferometry

1. Record blurred images:— where sk(x, y) is kth point-spread function due to atmospheric turbulence

gk (x,y ) = f (x, y )∗ sk (x ,y ), k = 1, . . ., K

A. Labeyrie, "Attainment of Diffraction Limited Resolution in Large Telescopes by Fourier Analysing Speckle Patterns in Star Images," Astron. and Astrophys. 6, 85-87 (l970).

2. Fourier transform:— where Sk(u, v) is kth optical transfer function

3. Magnitude square and average:

4. Measure or determine transfer function— atmospheric model or measure reference star

Gk (u,v ) = F (u,v )Sk (u,v ), k = 1, . . ., K

1K

Gk (u,v )2

k =1

K∑ = F (u,v )2 1

KSk (u,v )

2

k =1

K∑

1K

Sk (u,v )2

k =1

K∑

5. Divide by 1K

Sk (u,v )2

k=1

K∑ to get F (u,v )2 = squared Fourier magnitude

JRF -5

Image Reconstruction fromSimulated Speckle Interferometry Data

J.R. Fienup, "Phase Retrieval Algorithms: A Comparison," Appl. Opt. 21, 2758-2769 (1982).

Labeyri’sstellar speckleinterferometry

gives this

JRF -6

•  Fourier magnitude of astronomical object can come from–  Stellar speckle interferometry–  Michelson stellar (amplitude) interferometry–  Intensity interferometry (Hanbury Brown – Twiss)

•  For coherently illuminated objects: lensless imaging–  e.g. x-ray diffraction, laser remote sensing

•  From MRI, etc.,—Any imaging modality where when Fourier phase is lost

•  Have Fourier magnitude; Want to reconstruct an image from it

Reconstruction from Fourier Magnitude

JRF -7

Phase Retrieval Basics

Phase retrieval problem: Given F (u,v ) and some constraints on f (x,y ), Reconstruct f (x,y ), or equivalently retrieve ψ (u,v )

�

Fourier transform: F (u ,v ) = f (x ,y )e −i 2π (ux +vy )dxdy−∞∞∫∫

= F (u ,v )e iψ (u ,v ) = F f (x ,y )[ ]

�

Inverse transform: f (x ,y ) = F (u ,v )e i 2π (ux +vy )dudv−∞∞∫∫ = F −1 F (u,v )[ ]

(Inherent ambiguities: phase constant, images shifts, twin image all result in same data)

�

F (u,v ) = F f (x ,y )[ ] = F e ic f (x − xo ,y − yo )[ ] = F e ic f * (−x − xo ,−y − yo )[ ]

�

Autocorrelation:

rf (x ,y ) = f ( ′ x , ′ y )f * ( ′ x − x , ′ y − y )d ′ x d ′ y −∞∞∫∫ = F −1 F (u,v ) 2[ ]

• Patterson function in crystallography is an aliased version of the autocorrelation• Need Nyquist sampling of the Fourier intensity to avoid aliasing

JRF -8

Constraints in Phase Retrieval

•  Nonnegativity constraint: f(x, y) ≥ 0–  True for ordinary incoherent imaging, x-ray diffraction, MRI, etc.–  Not true for wavefront sensing or coherent imaging

•  The support of an object is the set of points over which it is nonzero–  Meaningful for imaging objects of finite extent on dark backgrounds–  Wavefront sensing through a known aperture–  Essential for complex-valued objects

•  Atomiticity when have angstrom-level resolution–  For crystals -- not applicable for coarser-resolution, single-particle

•  Object intensity constraint (wish to reconstruct object phase)–  E.g., measure wavefront intensity in two planes (Gerchberg-Saxton)–  If available, supersedes support constraint

•  Statistical information–  Maximum entropy, total variations, sparsity, sharpness, compactness,

weak object phase, smooth object phase, etc.

JRF -9

Phase is More Important than Amplitude

G

M

|FT[G]|

|FT[M]|

phase{FT[G]}

phase{FT[M]}

FT-1[|FT[G]| exp[i phase{FT[M]}]]

FT-1[|FT[M]| exp[i phase{FT[G]}]]

JRF -10

Is Phase Retrieval Possible?

1970’s: Some said, “Can’t be done. The solution won’t be unique.”

JRF -11

First Phase Retrieval Result

(a) Original object, (b) Fourier modulus data, (c) Initial estimate(d) – (f) Reconstructed images — number of iterations: (d) 20, (e) 230, (f) 600

Reference: J.R. Fienup, Optics Letters, Vol 3., pp. 27-29 (1978).

JRF -12

Form New InputUsing Image Constraints

Satisfy FourierDomain

Constraints

Constraints:Support,

(Nonnegativity)

Measured Data:Magnitude, |F|

g'

g G = |G| e iφ

START:Initial Estimate

{ }–1F G' = φi|F| e

{ }F

Measured intensity

Iterative Transform Algorithm

F

gk +1 x,y( ) = ′gk x,y( ) , ′gk x,y( ) satisfies constraintsgk x,y( ) − β ′gk x,y( ), ′gk x,y( ) violates constraints

⎧⎨⎩

Hybrid Input-Output version

F –1

= Gerchberg-Saxton algorithm if have

measured intensities in both domains

JRF -13

Error Reduction = Projection onto Sets

Constraint Set #1(non-convex)

Constraint Set #2(convex)

Start

a

b

If a, b ∈ S,then c = t a + (1 – t) b ∈ S, 0 ≥ t ≥ 1

S is convex: If a, b ∈S, then c = t a + 1− t( )b ∈S, 0 ≤ t ≤1

JRF -14

Error-Reduction and HIO

•  Error reduction algorithm

–  Satisfy constraints in object domain–  Equivalent to projection onto (nonconvex) sets algorithm–  Proof of convergence (weak sense)–  In practice: slow, prone to stagnation, gets trapped in local minima

•  Hybrid-input-output algorithm

–  Uses negative feedback idea from control theory•  β is feedback constant

–  No convergence proof (can increase errors temporarily)–  In practice: much faster than ER, can climb out of local minima

�

ER: gk +1 x( ) =′ g k x( ) , x ∈S & ′ g k x( ) ≥ 0

0 , otherwise⎧ ⎨ ⎩

�

HIO: gk +1 x( ) =′ g k x( ) , x ∈S & ′ g k x( ) ≥ 0

gk x( ) − β ′ g k x( ) , otherwise⎧ ⎨ ⎩

H. Takajo, T. Takahashi et al, J. Opt. Soc. Am. A (1997 & 1998)

JRF -15

Progression of Iterations

Initial support constraint

Initial estimate 1 iteration

5 iterations 20 iterations 25 iterations

35 iterations 45 iterations 55 iterations

JRF -16

Error Metric versus Iteration Number

JRF -17

Autocorrelation Support= Sum of Support Sets

S = x : f x( ) > 0{ }Object Support

aX + bY ≡ a x + b y : x ∈X and y ∈Y{ }Set addition (Minkowski sum) and multiplication

A = ∪

y∈SS − y( ) = S −S = x − y : x,y ∈S{ }

Autocorrelation support

In , f (y + x) is f (x) translated by –y

is a weighted sum of translated versions of f (x), weighted by f (y) rf x( )rf x( )

u, x may be multidimensional

Complex Fourier transform

F u( ) = f x( )e−i2πuxdx

−∞

∞

∫ = F f x( )[ ]∫Autocorrelation function

rf x( ) = f y( )f y − x( )dy−∞

∞∫∫ = f y( )f y + x( )dy−∞

∞∫∫ = F−1 F u( ) 2⎡

⎣⎤⎦

assuming f is real, nonnegative

A can have points missing if f is complex-valued or bipolar

•  Solve the inverse problem:Given A, find the smallest set L inside of which all S “fit”, where A = S – S

JRF -18

Bounds on Object Support

Object Support

Autocorrelation Support

Triple Intersection of Autocorrelation Supports

a1

a1

a2

a2

0

0

Triple-Intersection Rule: [Crimmins, Fienup, & Thelen, JOSA A 7, 3 (1990)]

Autocorrelation: rf (x,y ) = f ( ′x , ′y )f * ( ′x − x, ′y − y )d ′x d ′y−∞∞∫∫ = F−1 F (u,v ) 2⎡

⎣⎤⎦

JRF -19

Triple Intersection for Triangle Object

 • Family of solutions for object support from autocorrelation support

 • Use upper bound for support constraint in phase retrieval

(a) (b)

(c)

(d)

Object Support Autocorrelation Support

Triple Intersection – Support Constraint

AlternativeObject Support

JRF -20

Shrinkwrap1 Example

Initial Support Guess Optimize With Initial Support Guess

Blur, Threshold, and Update Support

[1] Marchesini, S., He, H., Chapman, H. N., Hau-Riege, S. P., Noy, A., Howells, M. R., Weierstall, U., and Spence, J. C. H., “X-ray image reconstruction from a diffraction pattern alone,” Phys. Rev. B 68, 140101 (2003).

JRF -21

•  Find set of optimization parameters, g (image pixel values), to minimize an objective function consisting of data consistency and prior knowledge

•  Data consistency (example)

Phase Retrieval by Nonlinear Optimization

ω i = weights on prior knowledge g x,y( ) = argmin

gΦdata g( ) + ω iΦprior,i g( )

i∑

⎧⎨⎪

⎩⎪

⎫⎬⎪

⎭⎪

|F |2 = measured Fourier intensity

Φdata g( ) = W u,v( ) FT g⎡⎣ ⎤⎦

2− F u,v( ) 2⎧

⎨⎩

⎫⎬⎭∫∫

2Amplitude only:

W = weighting (optional)

•  Example priors–  Non-negativity

–  Support

Φprior1 = g x,y( ) 2

x,y( )∈ g x,y( )<0{ }∑

Φprior2 = g x,y( ) 2

x,y( )∉S∑

Can also implement support by only varying values of g within S

S = points in support

JRF -22

Nonlinear Optimization Algorithms Employing Gradients

Minimize Error Metric, e.g.:

Repeat three steps:

1. Compute gradient:

     ∂E∂p1

, ∂E∂p2

, …

2. Compute direction ofsearch 3. Perform line search

a

b

c

Parameter 1

Para

met

er 2

Contour Plot of Error Metric

Gradient methods:(Steepest Descent)Conjugate GradientBFGS/Quasi-Newton …

E = W u( ) G u( )2 – F u( )2⎡

⎣⎤⎦2

u∑ , G u( ) = F g x( )[ ]

JRF -23

Analytic Gradients

P [•] can be a single FFT or multiple-plane Fresnel transforms

with phase factors and obscurations

Analytic gradients very fast compared with

calculation by finite differences J.R. Fienup, “Phase-Retrieval Algorithms for a Complicated Optical System,” Appl. Opt. 32, 1737-1746 (1993).J.R. Fienup, J.C. Marron, T.J. Schulz and J.H. Seldin, “Hubble Space Telescope Characterized by Using Phase Retrieval Algorithms,” Appl. Opt. 32 1747-1768 (1993).A.S. Jurling and J.R. Fienup, “Applications of Algorithmic Differentiation to Phase Retrieval Algorithms,” J. Opt. Soc. Am. A (June 2014).

g(x ) = gR x( ) + i gI x( ) = mo x( )eiθ x( ) , θ x( ) = ajZ j x( )j =1

J∑Optimizing over

where

GW u( ) =W u( ) F u( ) G(u)

F u( ) −G u( )⎡

⎣⎢

⎤

⎦⎥ , and gW (x ) = P† GW u( )⎡⎣ ⎤⎦

For point-by-point pixel (complex) value, g(x),∂E

∂g(x ) = 2 Im gW * (x ){ }

For point-by-point phase map, θ(x), ∂E∂θ(x )

= 2 Im g x( )gW * x( ){ }For Zernike polynomial coefficients, ∂E

∂aj = 2 Im g x( )gW * x( )

x∑ Z j x( )⎧

⎨⎩

⎫⎬⎭

E = W u( ) G u( ) – F u( )⎡⎣ ⎤⎦

2

u∑ , G u( ) = P g x( )[ ]

JRF -24

•  Object: electron density function–  Periodic, continuous

•  Measurements–  Fourier intensity, undersampled

•  Bragg peaks are half-Nyquist sampled–  Missing phase

•  Prior information–  Nonnegativity (usually)–  Atomiticity = object consists of a finite number of atoms within a unit cell

•  Just need to recover the 3-D locations and strengths of a number of electron peaks

•  To reconstruct an electron density function, need to retrieve the Fourier phase — solve the “phase retrieval” problem

•  An early form of computational imaging and of compressive sensing

X-ray Crystallography

JRF -25

•  29 Nobel Prizes associated with crystallographyhttp://www.iucr.org/people/nobel-prize

•  Bragg solved the first crystal structure, salt, in 1914²  Nobel Prize in Physics, 1915

"for their services in the analysis of crystal structure by means of X-rays”Sir William Henry Bragg & William Lawrence Bragg

•  Nobel Prize in Medicine, 1962²  "for their discoveries concerning the molecular structure of nucleic acids and its

significance for information transfer in living material” [DNA] Francis Harry Compton Crick, James Dewey Watson, and Maurice Hugh Frederick Wilkins

•  Nobel Prize in Chemistry, 1985² “for their outstanding achievements in the development of direct methods

for the determination of crystal structures”Herbert A. Hauptman and Jerome Karle

X-ray CrystallographyExamples of Notable Achievements

JRF -26

Image Reconstruction from X-Ray Non-Crystallographic Diffraction Intensity

CoherentX-ray beam

Target Detectorarray(CCD)

Nano-object(electron micrograph)

Far-field diffraction pattern (Fourier intensity)

micron size or smaller

J.N. Cederquist, J.R. Fienup, J.C. Marron, and R.G. Paxman, “Phase Retrieval from Experimental Far-Field Speckle Data,” Optics Lett. 13, 619-621 (August 1988).J. Miao, P. Charalambous, J. Kirz, and D. Sayre, “Extending the methodology of x-ray crystallography to allow imaging of micrometre-sized non-crystalline specimens,” Nature (London) 400, 342–344 (1999).H. Chapman et al., “High-resolution ab initio 3-D x-ray diffraction microscopy,” JOSAA 23, 1179-1200 (2006).

Computer

Image

JRF -27

FirstCoherent Diffractive Imaging Experiment

JRF -28

Ptychography (Transverse Translation Diversity)

•  Moving the sample with respect to a known illumination pattern can provide suitably diverse measurements–  Makes phase retrieval more robust to stagnation, noise and ambiguities

H. M. L. Faulkner and J. M. Rodenburg, “Movable aperture lensless transmission microscopy: a novel phase retrieval algorithm,” Phys. Rev. Lett. 93, 023903 (2004). M. Guizar-Sicairos and J. R. Fienup, “Phase retrieval with transverse translation diversity: a nonlinear optimization approach,” Opt. Express 16, 7264-78 (2008).

Plane-wave

Moveable object

Fourier intensity measurements (one for each of several mask positions)

JRF -29

Ptychography Reconstruction Example

M. Guizar-Sicairos et al., “High- throughput ptychography using Eiger - Scanning X-ray nano-imaging of extended regions,” Optics Express 22, 14859, June 10, 2014.

98 Mega-pixel phase image of an Eiger detector arraywith 45 nm resolution

JRF -30

Fourier Ptychography

G.  Zheng,  “Breakthroughs  in  Photonics  2013:  Fourier  Ptychographic  Imaging,”  IEEE  Photonics  J.  6,  (April  2014)  

JRF -31

Unconventional Pupil-Plane Imaging Active Conformal Thin Imaging System

•  Coherent illumination, sensed by a thin, conformal array of detectors

•  Light weight, small depth, day/night•  With no imaging optics, a wider

aperture fits on an airplane–  Giving finer resolution

•  Phase retrieval used to form image–  Illumination-pattern constraint

Conformal Arrayof DetectorsCoherent

Illuminator

DT

DT

Aircraft fusilage

Focal plane detector array

Imaging lens

Conventional,depth ~ DT

Aircraft fusilage

Fresnel Lenses

Discrete detectors

DpDT

Dp

ACTIS, depth ~ DP << DT

•  Real 2- D aperture or 1-D array with aperture synthesis

JRF -32

Solution to Support Constraint Issues:Illumination Pattern with Separated Parts

•  Uniqueness more likely for separated support (areas with zero in between) m  T.R. Crimmins and J.R. Fienup, "Uniqueness of Phase Retrieval for

Functions with Sufficiently Disconnected Support," J. Opt. Soc. Am. 73, 218-221 (1983)

•  Phase retrieval works particularly well for support constraints with separated parts m  J.R. Fienup, "Reconstruction of a Complex-Valued Object from the

Modulus of Its Fourier Transform Using a Support Constraint," J. Opt. Soc. Am. A 4, 118-123 (1987)

•  Parts separated in x and y give fringes in detected speckle pattern m  If one separated part was a delta-function, then would be a hologram m  Somehow makes phase more readily available

JRF -33

Image Reconstruction Example

Far-field speckle intensity Pattern at sensor array

Includes aperture weighting

Extended scene Illuminated scene

Reconstructed image

Laser illumination pattern

Illumination optics 1/4 diameter of receiver array + Hanning weighting

Complex-valued SAR Image

JRF -34

PROCLAIM 3-D Imaging Phase Retrieval with Opacity Constraint LAser IMaging

tunable laser

direct-detectionarray

collected data set

reconstructed object

phase retrieval algorithm

3-D FFT

initial estimate

from locator

set

ν1, ν2, . . . νn

M.F. Reiley, R.G. Paxman J.R. Fienup, K.W. Gleichman, and J.M. Marron, “3-D Reconstruction of Opaque Objects from Fourier Intensity Data,” Proc. SPIE 3170-09, pp. 76-87 (1997)

reflecting, opaqueobject is on

2-D manifoldembedded in3-D volume

– sparse in z !

JRF -35

Imaging Correlography

•  Get incoherent-image information from coherent speckle pattern

•  Estimate 3-D incoherent-object Fourier squared magnitude

–  Like Hanbury-Brown Twiss intensity interferometry

•  Easier phase retrieval: have nonnegativity constraint on incoherent image

•  Coarser resolution since intensity interferometry SNR lower

FI(u,v ,w )2 ≈ Dk (u,v ,w )− Io[ ]⊗ Dk (u,v ,w )− Io[ ] k (autocovariance of coherent speckle pattern)

JRF -36

Object for Laboratory Experiments

ST Object. The three concentric discs forming a pyramid can be seen as dark circles at their edges. The small piece on one of the two lower legs was removed before this photograph was taken.

JRF -37

3-D Laser Fourier IntensityLaboratory Data

Data cube:

1024x1024 CCD pixels x 64 wavelengths

Shown at right: 128x128x64 sub-cube

(128x128 CCD pixels at each of 64 wavelengths)

JRF -38

Coherent Image Reconstructed byITA from One 128x128x64 Sub-Cube

JRF -39

Imaging through Scattering Media

• Gather light scattered onto detector array, with or without lens

• Autocorrelate to get autocorrelation of object — equivalent to Fourier intensity

• Phase retrieval to reconstruct incoherent image

• Related to Labeyrie’s technique

JRF -40

Summary of Phase Retrieval

•  Phase retrieval can give diffraction-limited images–   Despite atmospheric turbulence or aberrated optics

–   Despite lack of Fourier or Fresnel phase measurements•  Can work for many imaging modalities

–   Passive, incoherent or laser imaging correlography— Nonnegativity constraint— Support constraint — may be loose and derived from given data

–  Active, coherent imaging — Complex-valued image, so NOT nonnegative

— Support constraint: must be tight and of special type — or helped by: 3-D, or reflective opaque object, or correlography

•  Can be used for wavefront sensing, using simple hardware

–  For atmospheric turbulence, telescope aberrations (HST, JWST),eye aberrations, optical metrology

–  Can be used for beam characterization (laser, x-ray)

–  Allows imaging system to be corrected for improved imagery

JRF -41

James Webb Space Telescope18 segments in folded-up configuration

need10’s of nmalignmentover 6.5 m