phaseless imaging algorithms and applications › materials › 2017-2018 › sw8.14-18.17 ›...
Post on 06-Jul-2020
1 Views
Preview:
TRANSCRIPT
JRF -1
Phaseless Imaging Algorithmsand Applications
James R. Fienup
Robert E. Hopkins Professor of OpticsUniversity of Rochester
Institute of Opticsfienup@optics.rochester.edu
Presented to:Institute for Mathematics and Applications
University of Minnesota
August 2017
JRF -2
• Introductory phase retrieval example– Astronomical imaging
• Mathematical background– The problem– Iterative transform algorithms– Nonlinear optimization algorithms
• Recent application examples• Lensless coherent optical imaging• Non-crystallographic (x-ray) diffraction
Outline
JRF -3
Passive Imaging of Astronomical/Space Objects
blurredimage
object
aberrated optical system
}
LickObservatoryNaval Prototype Optical Interferometer
JWST
• Solutions:– Hubble Space Telescope (2.4 m diameter), $2 B; JWST $8 B– Adaptive optics + laser guide star, $10’s M– Optical interferometry, $10’s M– Stellar speckle interferometry, << $1 M
• Problem: atmospheric turbulence causes phase errors, limits resolution– ≈ 1 arc-sec ≈ 5*10–6 rad. ≈ λ/ ro for λ = 0.5 μm and ro = 10 cm
• Best resolution as though through only 10 cm aperture (vs. Keck 10 m)
JRF -4
Labeyrie’s Stellar Speckle Interferometry
1. Record blurred images:— where sk(x, y) is kth point-spread function due to atmospheric turbulence
gk (x,y ) = f (x, y )∗ sk (x ,y ), k = 1, . . ., K
A. Labeyrie, "Attainment of Diffraction Limited Resolution in Large Telescopes by Fourier Analysing Speckle Patterns in Star Images," Astron. and Astrophys. 6, 85-87 (l970).
2. Fourier transform:— where Sk(u, v) is kth optical transfer function
3. Magnitude square and average:
4. Measure or determine transfer function— atmospheric model or measure reference star
Gk (u,v ) = F (u,v )Sk (u,v ), k = 1, . . ., K
1K
Gk (u,v )2
k =1
K∑ = F (u,v )2 1
KSk (u,v )
2
k =1
K∑
1K
Sk (u,v )2
k =1
K∑
5. Divide by 1K
Sk (u,v )2
k=1
K∑ to get F (u,v )2 = squared Fourier magnitude
JRF -5
Image Reconstruction fromSimulated Speckle Interferometry Data
J.R. Fienup, "Phase Retrieval Algorithms: A Comparison," Appl. Opt. 21, 2758-2769 (1982).
Labeyri’sstellar speckleinterferometry
gives this
JRF -6
• Fourier magnitude of astronomical object can come from– Stellar speckle interferometry– Michelson stellar (amplitude) interferometry– Intensity interferometry (Hanbury Brown – Twiss)
• For coherently illuminated objects: lensless imaging– e.g. x-ray diffraction, laser remote sensing
• From MRI, etc.,—Any imaging modality where when Fourier phase is lost
• Have Fourier magnitude; Want to reconstruct an image from it
Reconstruction from Fourier Magnitude
JRF -7
Phase Retrieval Basics
Phase retrieval problem: Given F (u,v ) and some constraints on f (x,y ), Reconstruct f (x,y ), or equivalently retrieve ψ (u,v )
�
Fourier transform: F (u ,v ) = f (x ,y )e −i 2π (ux +vy )dxdy−∞∞∫∫
= F (u ,v )e iψ (u ,v ) = F f (x ,y )[ ]
�
Inverse transform: f (x ,y ) = F (u ,v )e i 2π (ux +vy )dudv−∞∞∫∫ = F −1 F (u,v )[ ]
(Inherent ambiguities: phase constant, images shifts, twin image all result in same data)
�
F (u,v ) = F f (x ,y )[ ] = F e ic f (x − xo ,y − yo )[ ] = F e ic f * (−x − xo ,−y − yo )[ ]
�
Autocorrelation:
rf (x ,y ) = f ( ′ x , ′ y )f * ( ′ x − x , ′ y − y )d ′ x d ′ y −∞∞∫∫ = F −1 F (u,v ) 2[ ]
• Patterson function in crystallography is an aliased version of the autocorrelation• Need Nyquist sampling of the Fourier intensity to avoid aliasing
JRF -8
Constraints in Phase Retrieval
• Nonnegativity constraint: f(x, y) ≥ 0– True for ordinary incoherent imaging, x-ray diffraction, MRI, etc.– Not true for wavefront sensing or coherent imaging
• The support of an object is the set of points over which it is nonzero– Meaningful for imaging objects of finite extent on dark backgrounds– Wavefront sensing through a known aperture– Essential for complex-valued objects
• Atomiticity when have angstrom-level resolution– For crystals -- not applicable for coarser-resolution, single-particle
• Object intensity constraint (wish to reconstruct object phase)– E.g., measure wavefront intensity in two planes (Gerchberg-Saxton)– If available, supersedes support constraint
• Statistical information– Maximum entropy, total variations, sparsity, sharpness, compactness,
weak object phase, smooth object phase, etc.
JRF -9
Phase is More Important than Amplitude
G
M
|FT[G]|
|FT[M]|
phase{FT[G]}
phase{FT[M]}
FT-1[|FT[G]| exp[i phase{FT[M]}]]
FT-1[|FT[M]| exp[i phase{FT[G]}]]
JRF -10
Is Phase Retrieval Possible?
1970’s: Some said, “Can’t be done. The solution won’t be unique.”
JRF -11
First Phase Retrieval Result
(a) Original object, (b) Fourier modulus data, (c) Initial estimate(d) – (f) Reconstructed images — number of iterations: (d) 20, (e) 230, (f) 600
Reference: J.R. Fienup, Optics Letters, Vol 3., pp. 27-29 (1978).
JRF -12
Form New InputUsing Image Constraints
Satisfy FourierDomain
Constraints
Constraints:Support,
(Nonnegativity)
Measured Data:Magnitude, |F|
g'
g G = |G| e iφ
START:Initial Estimate
{ }–1F G' = φi|F| e
{ }F
Measured intensity
Iterative Transform Algorithm
F
gk +1 x,y( ) = ′gk x,y( ) , ′gk x,y( ) satisfies constraintsgk x,y( ) − β ′gk x,y( ), ′gk x,y( ) violates constraints
⎧⎨⎩
Hybrid Input-Output version
F –1
= Gerchberg-Saxton algorithm if have
measured intensities in both domains
JRF -13
Error Reduction = Projection onto Sets
Constraint Set #1(non-convex)
Constraint Set #2(convex)
Start
a
b
If a, b ∈ S,then c = t a + (1 – t) b ∈ S, 0 ≥ t ≥ 1
S is convex: If a, b ∈S, then c = t a + 1− t( )b ∈S, 0 ≤ t ≤1
JRF -14
Error-Reduction and HIO
• Error reduction algorithm
– Satisfy constraints in object domain– Equivalent to projection onto (nonconvex) sets algorithm– Proof of convergence (weak sense)– In practice: slow, prone to stagnation, gets trapped in local minima
• Hybrid-input-output algorithm
– Uses negative feedback idea from control theory• β is feedback constant
– No convergence proof (can increase errors temporarily)– In practice: much faster than ER, can climb out of local minima
�
ER: gk +1 x( ) =′ g k x( ) , x ∈S & ′ g k x( ) ≥ 0
0 , otherwise⎧ ⎨ ⎩
�
HIO: gk +1 x( ) =′ g k x( ) , x ∈S & ′ g k x( ) ≥ 0
gk x( ) − β ′ g k x( ) , otherwise⎧ ⎨ ⎩
H. Takajo, T. Takahashi et al, J. Opt. Soc. Am. A (1997 & 1998)
JRF -15
Progression of Iterations
Initial support constraint
Initial estimate 1 iteration
5 iterations 20 iterations 25 iterations
35 iterations 45 iterations 55 iterations
JRF -16
Error Metric versus Iteration Number
JRF -17
Autocorrelation Support= Sum of Support Sets
S = x : f x( ) > 0{ }Object Support
aX + bY ≡ a x + b y : x ∈X and y ∈Y{ }Set addition (Minkowski sum) and multiplication
A = ∪
y∈SS − y( ) = S −S = x − y : x,y ∈S{ }
Autocorrelation support
In , f (y + x) is f (x) translated by –y
is a weighted sum of translated versions of f (x), weighted by f (y) rf x( )rf x( )
u, x may be multidimensional
Complex Fourier transform
F u( ) = f x( )e−i2πuxdx
−∞
∞
∫ = F f x( )[ ]∫Autocorrelation function
rf x( ) = f y( )f y − x( )dy−∞
∞∫∫ = f y( )f y + x( )dy−∞
∞∫∫ = F−1 F u( ) 2⎡
⎣⎤⎦
assuming f is real, nonnegative
A can have points missing if f is complex-valued or bipolar
• Solve the inverse problem:Given A, find the smallest set L inside of which all S “fit”, where A = S – S
JRF -18
Bounds on Object Support
Object Support
Autocorrelation Support
Triple Intersection of Autocorrelation Supports
a1
a1
a2
a2
0
0
Triple-Intersection Rule: [Crimmins, Fienup, & Thelen, JOSA A 7, 3 (1990)]
Autocorrelation: rf (x,y ) = f ( ′x , ′y )f * ( ′x − x, ′y − y )d ′x d ′y−∞∞∫∫ = F−1 F (u,v ) 2⎡
⎣⎤⎦
JRF -19
Triple Intersection for Triangle Object
• Family of solutions for object support from autocorrelation support
• Use upper bound for support constraint in phase retrieval
(a) (b)
(c)
(d)
Object Support Autocorrelation Support
Triple Intersection – Support Constraint
AlternativeObject Support
JRF -20
Shrinkwrap1 Example
Initial Support Guess Optimize With Initial Support Guess
Blur, Threshold, and Update Support
[1] Marchesini, S., He, H., Chapman, H. N., Hau-Riege, S. P., Noy, A., Howells, M. R., Weierstall, U., and Spence, J. C. H., “X-ray image reconstruction from a diffraction pattern alone,” Phys. Rev. B 68, 140101 (2003).
JRF -21
• Find set of optimization parameters, g (image pixel values), to minimize an objective function consisting of data consistency and prior knowledge
• Data consistency (example)
Phase Retrieval by Nonlinear Optimization
ω i = weights on prior knowledge g x,y( ) = argmin
gΦdata g( ) + ω iΦprior,i g( )
i∑
⎧⎨⎪
⎩⎪
⎫⎬⎪
⎭⎪
|F |2 = measured Fourier intensity
Φdata g( ) = W u,v( ) FT g⎡⎣ ⎤⎦
2− F u,v( ) 2⎧
⎨⎩
⎫⎬⎭∫∫
2Amplitude only:
W = weighting (optional)
• Example priors– Non-negativity
– Support
Φprior1 = g x,y( ) 2
x,y( )∈ g x,y( )<0{ }∑
Φprior2 = g x,y( ) 2
x,y( )∉S∑
Can also implement support by only varying values of g within S
S = points in support
JRF -22
Nonlinear Optimization Algorithms Employing Gradients
Minimize Error Metric, e.g.:
Repeat three steps:
1. Compute gradient:
∂E∂p1
, ∂E∂p2
, …
2. Compute direction ofsearch 3. Perform line search
a
b
c
Parameter 1
Para
met
er 2
Contour Plot of Error Metric
Gradient methods:(Steepest Descent)Conjugate GradientBFGS/Quasi-Newton …
E = W u( ) G u( )2 – F u( )2⎡
⎣⎤⎦2
u∑ , G u( ) = F g x( )[ ]
JRF -23
Analytic Gradients
P [•] can be a single FFT or multiple-plane Fresnel transforms
with phase factors and obscurations
Analytic gradients very fast compared with
calculation by finite differences J.R. Fienup, “Phase-Retrieval Algorithms for a Complicated Optical System,” Appl. Opt. 32, 1737-1746 (1993).J.R. Fienup, J.C. Marron, T.J. Schulz and J.H. Seldin, “Hubble Space Telescope Characterized by Using Phase Retrieval Algorithms,” Appl. Opt. 32 1747-1768 (1993).A.S. Jurling and J.R. Fienup, “Applications of Algorithmic Differentiation to Phase Retrieval Algorithms,” J. Opt. Soc. Am. A (June 2014).
g(x ) = gR x( ) + i gI x( ) = mo x( )eiθ x( ) , θ x( ) = ajZ j x( )j =1
J∑Optimizing over
where
GW u( ) =W u( ) F u( ) G(u)
F u( ) −G u( )⎡
⎣⎢
⎤
⎦⎥ , and gW (x ) = P† GW u( )⎡⎣ ⎤⎦
For point-by-point pixel (complex) value, g(x),∂E
∂g(x ) = 2 Im gW * (x ){ }
For point-by-point phase map, θ(x), ∂E∂θ(x )
= 2 Im g x( )gW * x( ){ }For Zernike polynomial coefficients, ∂E
∂aj = 2 Im g x( )gW * x( )
x∑ Z j x( )⎧
⎨⎩
⎫⎬⎭
E = W u( ) G u( ) – F u( )⎡⎣ ⎤⎦
2
u∑ , G u( ) = P g x( )[ ]
JRF -24
• Object: electron density function– Periodic, continuous
• Measurements– Fourier intensity, undersampled
• Bragg peaks are half-Nyquist sampled– Missing phase
• Prior information– Nonnegativity (usually)– Atomiticity = object consists of a finite number of atoms within a unit cell
• Just need to recover the 3-D locations and strengths of a number of electron peaks
• To reconstruct an electron density function, need to retrieve the Fourier phase — solve the “phase retrieval” problem
• An early form of computational imaging and of compressive sensing
X-ray Crystallography
JRF -25
• 29 Nobel Prizes associated with crystallographyhttp://www.iucr.org/people/nobel-prize
• Bragg solved the first crystal structure, salt, in 1914² Nobel Prize in Physics, 1915
"for their services in the analysis of crystal structure by means of X-rays”Sir William Henry Bragg & William Lawrence Bragg
• Nobel Prize in Medicine, 1962² "for their discoveries concerning the molecular structure of nucleic acids and its
significance for information transfer in living material” [DNA] Francis Harry Compton Crick, James Dewey Watson, and Maurice Hugh Frederick Wilkins
• Nobel Prize in Chemistry, 1985² “for their outstanding achievements in the development of direct methods
for the determination of crystal structures”Herbert A. Hauptman and Jerome Karle
X-ray CrystallographyExamples of Notable Achievements
JRF -26
Image Reconstruction from X-Ray Non-Crystallographic Diffraction Intensity
CoherentX-ray beam
Target Detectorarray(CCD)
Nano-object(electron micrograph)
Far-field diffraction pattern (Fourier intensity)
micron size or smaller
J.N. Cederquist, J.R. Fienup, J.C. Marron, and R.G. Paxman, “Phase Retrieval from Experimental Far-Field Speckle Data,” Optics Lett. 13, 619-621 (August 1988).J. Miao, P. Charalambous, J. Kirz, and D. Sayre, “Extending the methodology of x-ray crystallography to allow imaging of micrometre-sized non-crystalline specimens,” Nature (London) 400, 342–344 (1999).H. Chapman et al., “High-resolution ab initio 3-D x-ray diffraction microscopy,” JOSAA 23, 1179-1200 (2006).
Computer
Image
JRF -27
FirstCoherent Diffractive Imaging Experiment
JRF -28
Ptychography (Transverse Translation Diversity)
• Moving the sample with respect to a known illumination pattern can provide suitably diverse measurements– Makes phase retrieval more robust to stagnation, noise and ambiguities
H. M. L. Faulkner and J. M. Rodenburg, “Movable aperture lensless transmission microscopy: a novel phase retrieval algorithm,” Phys. Rev. Lett. 93, 023903 (2004). M. Guizar-Sicairos and J. R. Fienup, “Phase retrieval with transverse translation diversity: a nonlinear optimization approach,” Opt. Express 16, 7264-78 (2008).
Plane-wave
Moveable object
Fourier intensity measurements (one for each of several mask positions)
JRF -29
Ptychography Reconstruction Example
M. Guizar-Sicairos et al., “High- throughput ptychography using Eiger - Scanning X-ray nano-imaging of extended regions,” Optics Express 22, 14859, June 10, 2014.
98 Mega-pixel phase image of an Eiger detector arraywith 45 nm resolution
JRF -30
Fourier Ptychography
G. Zheng, “Breakthroughs in Photonics 2013: Fourier Ptychographic Imaging,” IEEE Photonics J. 6, (April 2014)
JRF -31
Unconventional Pupil-Plane Imaging Active Conformal Thin Imaging System
• Coherent illumination, sensed by a thin, conformal array of detectors
• Light weight, small depth, day/night• With no imaging optics, a wider
aperture fits on an airplane– Giving finer resolution
• Phase retrieval used to form image– Illumination-pattern constraint
Conformal Arrayof DetectorsCoherent
Illuminator
DT
DT
Aircraft fusilage
Focal plane detector array
Imaging lens
Conventional,depth ~ DT
Aircraft fusilage
Fresnel Lenses
Discrete detectors
DpDT
Dp
ACTIS, depth ~ DP << DT
• Real 2- D aperture or 1-D array with aperture synthesis
JRF -32
Solution to Support Constraint Issues:Illumination Pattern with Separated Parts
• Uniqueness more likely for separated support (areas with zero in between) m T.R. Crimmins and J.R. Fienup, "Uniqueness of Phase Retrieval for
Functions with Sufficiently Disconnected Support," J. Opt. Soc. Am. 73, 218-221 (1983)
• Phase retrieval works particularly well for support constraints with separated parts m J.R. Fienup, "Reconstruction of a Complex-Valued Object from the
Modulus of Its Fourier Transform Using a Support Constraint," J. Opt. Soc. Am. A 4, 118-123 (1987)
• Parts separated in x and y give fringes in detected speckle pattern m If one separated part was a delta-function, then would be a hologram m Somehow makes phase more readily available
JRF -33
Image Reconstruction Example
Far-field speckle intensity Pattern at sensor array
Includes aperture weighting
Extended scene Illuminated scene
Reconstructed image
Laser illumination pattern
Illumination optics 1/4 diameter of receiver array + Hanning weighting
Complex-valued SAR Image
JRF -34
PROCLAIM 3-D Imaging Phase Retrieval with Opacity Constraint LAser IMaging
tunable laser
direct-detectionarray
collected data set
reconstructed object
phase retrieval algorithm
3-D FFT
initial estimate
from locator
set
ν1, ν2, . . . νn
M.F. Reiley, R.G. Paxman J.R. Fienup, K.W. Gleichman, and J.M. Marron, “3-D Reconstruction of Opaque Objects from Fourier Intensity Data,” Proc. SPIE 3170-09, pp. 76-87 (1997)
reflecting, opaqueobject is on
2-D manifoldembedded in3-D volume
– sparse in z !
JRF -35
Imaging Correlography
• Get incoherent-image information from coherent speckle pattern
• Estimate 3-D incoherent-object Fourier squared magnitude
– Like Hanbury-Brown Twiss intensity interferometry
• Easier phase retrieval: have nonnegativity constraint on incoherent image
• Coarser resolution since intensity interferometry SNR lower
FI(u,v ,w )2 ≈ Dk (u,v ,w )− Io[ ]⊗ Dk (u,v ,w )− Io[ ] k (autocovariance of coherent speckle pattern)
JRF -36
Object for Laboratory Experiments
ST Object. The three concentric discs forming a pyramid can be seen as dark circles at their edges. The small piece on one of the two lower legs was removed before this photograph was taken.
JRF -37
3-D Laser Fourier IntensityLaboratory Data
Data cube:
1024x1024 CCD pixels x 64 wavelengths
Shown at right: 128x128x64 sub-cube
(128x128 CCD pixels at each of 64 wavelengths)
JRF -38
Coherent Image Reconstructed byITA from One 128x128x64 Sub-Cube
JRF -39
Imaging through Scattering Media
• Gather light scattered onto detector array, with or without lens
• Autocorrelate to get autocorrelation of object — equivalent to Fourier intensity
• Phase retrieval to reconstruct incoherent image
• Related to Labeyrie’s technique
JRF -40
Summary of Phase Retrieval
• Phase retrieval can give diffraction-limited images– Despite atmospheric turbulence or aberrated optics
– Despite lack of Fourier or Fresnel phase measurements• Can work for many imaging modalities
– Passive, incoherent or laser imaging correlography— Nonnegativity constraint— Support constraint — may be loose and derived from given data
– Active, coherent imaging — Complex-valued image, so NOT nonnegative
— Support constraint: must be tight and of special type — or helped by: 3-D, or reflective opaque object, or correlography
• Can be used for wavefront sensing, using simple hardware
– For atmospheric turbulence, telescope aberrations (HST, JWST),eye aberrations, optical metrology
– Can be used for beam characterization (laser, x-ray)
– Allows imaging system to be corrected for improved imagery
JRF -41
James Webb Space Telescope18 segments in folded-up configuration
need10’s of nmalignmentover 6.5 m
top related