The Career of Lee Holeva

Imaging ScientistNovember 11, 2005Updated August 4, 2010

Selected Topics

• Determining Interferometer Fringe Count Error• Orbital Smear Modeling• Contrast Matching of Images• Estimation of DSL Line Speed using Neural

Networks• High Speed Sorting of Magazines using KIX

codes• Range from Camera Blur using Regularization

Fringe Count Error Estimation

• Fourier Transform interferometry measures scene spectra by computing the Fourier Transform of the interferogram of an imaged scene.

• A shift of the interferogram implies a phase shift of the complex spectrum:

• As the interferometer operates in the IR spectrum observed spectra will have both internal and external components.

njeSnxI 2)()(

A Phase Function

• We need a phase function characterizing just the phase change due to fringe count error.

• Define:

fringek

Externalk

Internalk

Externalk

Internalk

fringek

j

kk

kk

Externalk

InternalkHot

k

Internalk

Hotk

Earthk

kkkk

jHotk

Internalk

Coldk

Hotk

Coldk

k

jHotk

Internalk

Hotk

Earthkj

Coldk

Hotk

Earthk

k

eQjU

PR

A

A

A

AQPPU

eA

A

SS

SQ

eA

A

A

Ae

SS

SP

Im

cosIm2

2*

Phase to Fringe Count Error

• Given the sign of the square root the phase is calculated from R:

• The slope of the phase with respect to frequency shows the number of fringe count errors:

k

kfringek R

R

Re

Imtan 1

mirrorfringe vddfd

n/2

What about the sign of the Square Root?

• The linear relationship of the phase slope to fringe count error assumes a long optical path length, OPD. The phase can vary greatly from a straight line at both the beginning and end of the range of the interferogram travel.

• An adaptive algorithm is used to pick the best choice of square root sign.

Picking the sign using RLS

• For each frequency k there are two possible phases, that found using a positive square root sign and that found using a negative square root sign:

• The RLS algorithm may be used to adapt the square root sign by iterating over frequency:

kand

k

The Recursion

otherwisek

kkkkifkk

kxkwk

orw

kgkerrorkwkw

kPkxkgkPkP

kxkPkx

kxkPkg

kxkwkkerror

fkx

t

t

t

t

k

)1(

)1(ˆ)1()1(ˆ)1()1()1(

)1()()1(ˆ

0

)0(

0

)0()0(

)()()1()(

)1()()()1()(

)()1()(1

)()1()(

)()1()()(

1)(

Extrapolated Phase versus True Phase

Variation of Square Root Signs

Modeling Orbital Smear

• Satellites typically employ line scan sensors with time delay integration, TDI

• Due to the motion of the satellite, motion blur, orbital smear, occurs both along the array, the cross scan, and along the line of scan, the along scan direction.

• The modeling of orbital smear requires knowledge of the satellite’s orbit, its attitude, and the orientation of the imaging array.

The Satellite in Orbit

ECI frame (at the Earth’s center)

North Pole

Vernal Equinox

Eccentricity

PQW frame in the plane of the orbit

Orbit of the satellite

Line of Nodes (descending to ascending)

Constant angular momentum

Right Ascension of the Ascending Node

Angle of Perigee

Inclination Angle

A sequence of Frame Conversions

• Use Kepler’s third law to convert time along orbit to anomaly

• Use Kepler’s first law to get the position and velocity in the plane of the orbit

• Apply a sequence of rotations to go to inertial coordinates

• Take into account the rotation of the Earth to go to Earth Centered coordinates

earthperigeeapogee

perigeeapogee

earthperigeeearth

radiusAltAlt

AltAlte

radiusAlt

en

e

eE

EeEtn

2

1

2tan

1

1tan2

sin

3

1

eradiusAltp

rtzrotr

rzrotixrotzrotr

e

p

e

pr

earthperigee

ECIearthECF

PQWECI

PQW

1

),(

),(),(),(

0cos1

sin

cos1

cos

Slew the Satellite

• From a sequence of quaternions determine the attitude of the satellite

• At each time step during the scan locate the target by intersecting the line of sight vector with the surface of the Earth

4

13

12

13

23

41313131324

0

0

0

22

q

qquaternion

qq

qq

qq

qqqIqqqZYX ttpayload

slantranges

ZsrT payload

Cross Scan Smear is Proportional to the rate of change of GSD

• Smear along the array is proportional to fractional rate of change of G

• Assuming that the rotation rates are chosen so that the TDI stages remain perpendicular with the ground track, the smear decomposes into the sum of a range term, due to changes of slant range, and a zenith term, due to rotations about the zenith vector. Even without the parallel TDI assumption, a first order analysis indicates that the sum of range and zenith terms remains at least approximately valid.

Y

Z

Unit Target Zenith

TXZ

TXIssIG

sin

sI

I is the instantaneous field of view and s is the slant range

Along Scan Smear is proportional to changes of the

rotation rates• Along scan smear, along the ground track, is

proportional to changes of both the yaw and pitch rates. However, the if the condition of the TDI stages remaining perpendicular to the ground track is maintained then the along scan smear is minimized. Changes of terrain altitude will however induce changes of the rotation rates and consequentially nonzero along scan smear.

Conclusions

• Two simulators were developed in Matlab, the first kept the satellite fixed in position allowing the variation of the free parameters of scanning to ascertain the variations of orbital smear.

• The second simulator moved the satellite along its orbit producing smear images of detector position versus time.

• Critical trade study questions may now be easily answered.

Setting Up a Delta NIIRS Image Quality Experiment

Common Source Image

Image simulator

Test MTF

Image simulator

Marker MTF

Wiener Filter

Wiener Filter

Test Enhancement Kernel

Marker Enhancement Kernel

Marker Image

Test Image

bmcm

Tonal transfer curves

bt ct

Tonal transfer curves

The Marker and Test images must have the same Eye Scale

and Contrast• Eye scale is easily

matched by interpolation

• To get the image contrast to agree requires adjustment of the the dynamic range adjustment adder and multiplier and the selection of the appropriate tonal transfer curve

cbIFI inkout

Nonlinear Optimization used to Match Contrast

• Key Observation: the tonal transfer curves are smooth and hence derivatives may be calculated (say by the use of a 2nd order Butterworth filter)

• Setup the contrast matching problem as the minimization of:

2

ker2

1cbIFIJ testkmar

Only a Few Iterations are needed for each Image pair

• The well known method of Levenberg-Marqurdt may be used to perform the minimization

• To both speed up the process and to desensitize the match to noise, perform a block average or median followed by a decimation

Conclusions

• A dramatic increase in productivity results: A part of the experiment setup effort that previously took weeks, if not months, is now reduced to a few hours.

• The contrast matching may be embedded into the shell scripts (it is possible to call Matlab commands directly from the UNIX command line without the use of a GUI) used to create the marker and test images.

Estimation of DSL Line Speed using Neural Networks

• A Pair of Hybrid RBF/MLP neural networks are trained to predict the maximum downstream and upstream speeds of copper telephone lines

• Lines are characterized by impedance and capacitance measurements made over frequency acquired from telephone lines of varying length with and without bridge taps.

• The data tends to contain significant correlations. Techniques such as the Karhunen-Loeve transform are employed to truncate the data set into its most significant parts.

Hybrid RBF/MLP Neural Net

Training the Net: Conditioning the Features

• Line speed is the linguistic variable. Different basis units for lines of differing speeds should make speed estimation easier. Condition the location of basis units by fuzzy sets:

Training the Net: Conditional Fuzzy Clustering

• Classical K-means:

• Fuzzy C-means:

• Conditional Fuzzy C-means:

• The conditioning terms, fk, come from the predefined triangular fuzzy sets

n

k

c

iik vxdVUJ

1 1

2 ,),(

ik

n

k

c

i

mik vxduVUJ

,),( 2

1 1

11

1

1

1

1

2

m

fu

u

xuv

dd

fu

c

ikik

n

k

mik

n

kk

mik

i

c

j

m

jk

ik

kik

Training the Net: Size and Location of the Basis Units

• The following rule is suggested: features having a cluster membership near one should be reside near the basis unit center while features with very small cluster membership values should be reside far from the basis unit center. This idea is captured by the Mahalobis distance between the feature vector and the cluster center.

• The centers of the basis units and the center of the clusters coincide.

• Using Gaussian RBF units, the width of the basis units is characterized by the covariance matrix. For this work, off diagonal correlations are ignored. A number of ad-hoc algorithms are possible to produce basis units that mimic the trained clusters.

Training the Net: Training the Output Layer

• Gaussian units form the basis layer:

• The predicted speed is formed by a vector matrix vector multiply:

• Either the standard Levenberg-Marquardt algorithm or a regularized version may be used to train the output matrix. The use of the full feature vector on the feed-forward path may result in over fitting.

2

21

1

11

111

100

00

001

expexp

iM

i

i

cct

ct

w

wC

vxCvxvxCvxb

xx

xWbspeed

reduced

reducedt

Speed Predictions Plotted against the Actual Maximum Speed

High Speed Sorting of Magazines using KIX codes

The behavior of the sorting machine is Postal code

Dependent• Magazines move down a conveyor belt at a rate of about

30 magazines per minute

• The destination of the sort is dependent upon the postal codes

• Each magazine may have up to three postal codes at any orientation or position

• For adequate resolution the images may be need to be quite large, 3000 by 4000 pixels is typical

• The only constraint is that a minimum background contrast is guaranteed.

An adaptive threshold is employed to binarize the image

A Hit-or-Miss Transform scans the image for Candidates

• The Hit-or-Miss transform uses masking coupled with logical rules to detect regular features in the binary image. Scores from the Hit-or-Miss are then summed within a tiling of small windows.

Candidate Orientation by Performing an Eigenfit

• Assuming that the position is known, say by the candidate’s centroid, the problem of determining the orientation may be stated as fitting a subspace. For the problem at hand, this means finding the line having minimum orthogonal distance to any point of the candidate.

• To turn this idea into a real algorithm, setup the Lagrangian and take the derivative with respect to the subspace vector:

1,0

sincos0

2

uwhereXu

or

yx

t

02

1

1

1

uIxxu

L

uuuxxuL

M

i

tii

tM

i

tii

t

u is the eigenvector associated with the smallest eigenvalue of the scatter matrix

Is the Candidate really a Postal Code?

• Having rotated the candidate to the horizontal, setup a feature vector as the discrete fourier transform of one channel of wavelet coefficients

• Assuming Gaussian statistics, a simple maximum likelihood classifier, with both the mean vector and covariance matrix estimated from positive postal code examples works very well.

postalcode

thresholdmxCmxxd t

1

2

1

Not a Postal Code

A Postal Code

Use Fuzzy Logic to Classify Bars

The Run-Time Architecture

Two stage pipeline (2 magazines processed at a time)

Conclusions

• After developing the imaging algorithms on a Sparc station the Hit-or-Miss operator was implemented in a Gate-Array and the C code ported to the C40 DSPs by a contractor. I then continued the development by debugging the delivered code.

• The magazine sorter ran at full speed with a very small error rate.

• Most problems were not with the imaging, but with mechanical aspects of the machine.

The Inverse Problem of Range from Camera Blur

Two Solutions

• Frequency Domain, blur from the estimated transfer function:

• Spatial Domain, blur from the estimated local point spread function:

22

2

1GHHXD

NEcamera

4

11

2))(()()()()()(

kkk

n

i

tincamera nwcnwiuidinE

Perceived Depth of a Block of Steps

Summary

• I’ve been involved in a number of challenging projects which have included modeling and simulation, satellite system engineering, the development of signal and image processing algorithms, nonlinear estimation, optics, and real-time systems.

• Also, a copy of my paper on mean field annealing, that appeared in the Journal of Electronic Imaging (April, 1996), may be found at (in two parts):

– http://www.legiotricesima.org/netpaper1-15.pdf

– http://www.legiotricesima.org/netpaper16-31.pdf