-
8/9/2019 First Electro Optical System of the 21st Century
1/8
PROCESSING REQUIREMENTS FOR THE FIRST ELECTRO OPTIC
SYSTEM OF THE TWENTY FIRST CENTURY
William F. O'Neil
Northrop Grumman Corporation
Electronic Sensors and Systems Division
0NEIL.W.FBPOSTAL.ESSD.NORTHGRUM.COM
ABSTRACT
The Distributed Aperture Infrared System (DAIRS)
defines a new direction in the design of electro-optic
systems. A multiplicity of identical sensors provides
the data for a central processing system. A variety
of signal processing algorithms are used to obtain
the functionality that was previously obtained using
specialized sensors for each function. This permits
a single system to be a Missile Warning set, an
Infrared Search and Track set, the Pilotage sensor,
and a Situational Awareness system as well as
performing various tasks associated with the
warfighting mission of the platform. This paper
describes the DAlRS system being evolved for the
Joint Strike Fighter (JSF) which will be the first
US
aircraft developed in the 21 century. The system is
uniquely processor driven, representing a transition
in electro-optics that parallels the transition in radar
systems that occurred
20-30
years ago. The
algorithms used to implement the functions are
briefly described. The paper presents processor
loading,
I/O
and memory requirements peculiar to
the JSF. Application of the DAlRS technology to
other platforms and applications is also discussed.
A short speculative section on possible future trends
concludes the paper.
BACKGROUND
The development of electro-optic sensors for both
the visible and infrared IR) spectra has been a
major thrust of military systems for the last thirty
years. The third generation of
IR
sensors is now
becoming available. These are two-dimensional
arrays of detectors providing as many as one million
detectors in a single sensor. These sensors alter
the economics of IR sensing in the same way that
the Silicon CCD changed the visible sensor market.
A basic IR sensor can now consist of optics, the
detector array, and some simple electronics. For
the highest performance systems, a cryogenic,
closed cycle cooler
is
still required. The emergence
of room temperature IR detectors will further simplify
the sensors and reduce the cost. These new
sensors provide an opportunity to rethink the design
of E-0 systems. At Northrop Grumman, we have
been pursuing the development of new system
concepts [1,2] since the first staring sensor concepts
were being investigated. DAlRS is a program
funded jointly by Northrop Grumman and the US
Navy Naval Air Warfare Center to bring a distributed
aperture system to flight test status.
DAIRS CONCEPT
ThreaWisslle
Wsrni
Figure 1 DAlRS Provides
4n
Steradian Sensor
Coverage and Multiple Functions
DAlRS (Figure 1) uses an array of sensors,
strategically located around the aircraft to provide
47c
sensing for missile threat warning, IR aircraft search
and track, situational awareness, pilotage, battle
damage assessment, and weapon delivery support.
These sensors are fixed to the aircraft which avoids
the high cost of a mechanism for pointing and
stabilization. DAlRS provides the functions
previously assigned to separate sensor systems with
a single system. A typical sensor design (Figure 2)
is less than five percent of the weight, volume and
power
of
current airborne
IR
sensors. A suite of six
sensors is the minimum configuration for full
coverage due to limitations in the design of optics
with a field of view of more than about 90x900.
0-7803-4150-3/97 10.00 997
l
5.1 15
http://postal.essd.northgrum.com/http://postal.essd.northgrum.com/ -
8/9/2019 First Electro Optical System of the 21st Century
2/8
Figure
2
Projected DAldS Production
Configuration fits in
150
in3 volume
The functions that DAlRS provides are enhanced if
the resolution is as fine as the optics will permit.
Current wide field of view optic designs can support
detector arrays with as many as 4Kx4K detectors
which is well beyond the near term detector state of
the art. For the Joint Strike Fighter
(JSF)
which is
the next opportunity for an advanced
E-0
system,
the largest arrays that will be available for the E&MD
phase will have l K x l K detectors which have begun
the development cycle. When JSF enters
production in 2006, these arrays will be considered
standard COTS items.
To meet the pedormance goals of the
JSF,
DAlRS
includes a substantial signal processing component
that will increase the effective resolution of the
sensors by at least
2X.
If sufficient processing
power is available, a 4X resolution increase is
possible within the physical limits of optics and
detectors. Figure
3
[3]
illustrates the resolution
enhancements that have been demonstrated using a
technique called microscanning.
b
4:l microscan
Figure
3
M ~ c r ~ ~ ~ ~ n n ~ n gncreases Effective
Resolution to Optical Limit
The results shown in Figure 3were obtained using a
Maximum Likelihood procedure that combined the
samples from sixteen images, then estimated
underlying sample values on a sampling grid w
was four times denser in each direction than
original sampling grid. The estimation process
iterated about fifty times
to
obtain converg
The procedure required hundreds
of
operation
output sample, and there are 16 output sample
input sample.
A
technique has been develop
Northrop Grumman that requires less
than
operations per output sample. Determinati
line of sight motion from a moving platfor
necessary and requires about 80 operations
input sample.
In addition
to
resolution enhancement, DAlR
addressing the non-uniformity correction (NUC
the detector outputs. Non-uniformity from det
to detector
is
a function of operating cond
including cryogenic temperature, scene tempera
atmospheric conditions as well as inh
differences between detectors including lea
currents, readout linearity and unit cell sele
switches. NUC
is
central to achieving max
range performance for the Missile Warning and
RST
functions. NUC, when combined with tem
frame integration can provide as much as a
improvement in sensitivity (Figure 4)[4]. The N
Equivalent Temperature Difference is plotted ag
background temperature under laboratory cond
to illustrate one aspect
of
the NUC problem.
constant offset term is coherently integrated w
temporal integration is used. For the two-
calibration scheme used in this COTS cam
temporal integration is only effective at
calibration temperatures, with offset e
dominating the noise results elsewhere.
1
1 I
I
I
E I I I I
I I I
I
0.1
0.01
O . O O I
15
20 25
30
35
ackground Temperature- O C
Figure 4- NUC Enables Temporal Integratio
Improve Sensitivity
NUC is included in the microscan operation by u
the microscan
to
determine the spatial s
derivative which is then integrated to recove
5.1 16
-
8/9/2019 First Electro Optical System of the 21st Century
3/8
scene. This operation suppresses the offset error
which is the important limiter of sensitivity. Gain
errors are both more stable and also less critical in
limiting the detection of low signal to noise targets.
Gain corrections are performed using factory
calibrations and are assumed constant over sensor
life. NUC computation cost is operations per input
sample. Temporal integration requires
4
operations
per output sample.
Providing effective temporal integration on a dynamic
platform requires that successive frames be
precisely registered. Misregistration causes
smearing of the image with a loss of high spatial
frequency detail. The combination of resolution
enhancement, NUC and temporal integration for
sensitivity enhancement requires that the image
motion must be determined to a small fraction of the
smallest scene details to be recovered. The motion
resolution required is in the range of 0.01 to
0.05
detector instantaneous field of view or about 10-50
PPM of the sensor field of view for a sensor with a
1Kx lK focal plane. The problem of registering
successive images has been extensively studied
[e.g. 5, 6, 71.
DAlRS Applications
DAlRS applications can be classed broadly as
imaging applications where the objective is to
maximize the detail, and detection of unresolved
targets such as missiles or aircraft at long ranges,
where sensitivity and suppression of interfering
clutter are the primary objectives. The imaging
applications include pilotagehavigation, situational
awareness, context support for targeting and battle
damage assessment. The unresolved target
applications include missile threat warning and IRST.
Imaging Applications
For pilotage applications, the interaction of DAlRS
with the display system and pilot determines the
requirements. Resolution at unity magnification is
needed to provide the pilot with an out the window
environment that is sufficient for operation
equivalent
to
VFR conditions. Based on experience
with navigation
IR
sensors currently in use, a
resolution of
0.5-1.0
milliradians appears
to
be the
minimum acceptable range. Achieving human
resolution limit (0.15 milliradians) may not be
necessary due to degradations associated with
aircraft vibration effects
on
pilot vision. A head
mounted display (HMD) is very desirable to
overcome the limits of cockpit display field of regard.
To avoid artifacts when using the HMD, it is
important to have a seamless image with little or no
distortion. A novel mode that will be provided with
DAIRS and an HMD is the ability to see through the
floor when landing a JSF in the VSTOL mode. In
addition, the pilot will be able to view nearby aircraft
for increased situational awareness using an
electronic steerable rear view mirror.
If the HMD is a transparent type that permits seeing
the actual scene with the sensed scene overlaid,
then precision registration is also mandated. The
planned approach for DAlRS is to remove all
distortions by projecting the pixels onto a standard
grid that is aligned with the aircraft IMU
to
a precision
of a fraction of the output sample resolution. This
requires that image warping algorithms operate in
real time (6 operations per output sample). Since
the sensors will be moving relative
to
the IMU due to
aircraft flexing in flight, the alignment properties must
also be calculated in real time. Rate sensors at
each DAIRS sensor are provided for that purpose.
The standard grid also provides registration between
the DAlRS image and the magnified image that is
provided by the targeting sensor. Insetting a portion
of the magnified image into the unity magnification
image provides the context for the magnified image
to aid the pilot in acquisition and identification of
targets.
Battle damage assessment (BDA) is best done with
a very high resolution sensor. However, DAlRS can
provide useful data and compensates for its limited
resolution by providing continuous, all-aspect
coverage. Events where time history is an important
clue include impact explosions, delayed detonation
weapons that produce an ejected plume of hot gas
or debris, and targets that burn after weapon impact.
For these cases, DAlRS is a functional BDA sensor
that can replace high resolution sensors. The key
requirement is sufficient dynamic range to permit
sensing these high intensity but brief events without
the overload and long recovery time of presently
fielded systems. Novel control strategies are
anticipated to optimize the use of DAlRS for BDA.
They will serve to augment the dynamic range of
60-
70
dB available on an intrascene basis
to
achieve an
interscene dynamic range greater than
90
dB.
Unresolved Target Applications
For
unresolved targets the issues are similar for both
missile warning and IRST. The maximum
achievable range at which a targethhreat can be
detected is determined by sensitivity. This is known
5.1 17
-
8/9/2019 First Electro Optical System of the 21st Century
4/8
as the clear sky range and is inversely proportional
to detector instantaneous field of view (IFOV).
Temporal integration increases this range as the
fourth root of the number of frames of integration
used. Since the target has not yet been detected,
and will usually be moving with respect
to
the inertial
background, direct target registration is not feasible.
Instead, an array velocity filter is used where
integration is performed for a range of target line of
sight (LOS) rates. For missile targets, the threats
are limited to objects that have a near zero LOS rate.
For aircraft targets, long range detection also limits
the LOS rate while advantage can be taken of the
low vertical range of angles and rates of climb
possible. Thus, while velocity filtering can be
computationally ntensive, the problem can be limited
by the physicsof each engagement.
The clear sky range is frequently of academic
interest because the available range is limited by
false alarms due to clutter. While
a
signal to noise
ratio of
3-5
might suffice for an acceptable false
alarm rate
for
a clear sky target, anecdotal reports
indicate that a typical aircraft against an urban clutter
background may require that the threshold for signal
to noise ratio be increased
to
50-100 to achieve an
acceptable false alarm rate. The resulting drop in
range
3-6x)
is dramatic, and nullifies the benefits of
a practical sensor design. Similar effects have been
noted for the missile threat case. Clutter
suppression is the key issue in the detection of
unresolved targets. The available strategies for
clutter suppression include spatial filters, track file
filters, velocity filters, and multi-spectral filters.
The characteristics of ground clutter are quite
variable. Measurement reported in [8] how a
general tendency to reduced clutter amplitude with
increased spatial frequency. This translates to
improved clutter suppression with increased sensor
resolution (e.g. Figure 5which uses the measured
trends from
[a]
for urban clutter). Spatial filtering is
used to exploit this characteristic as a first step in
most clutter suppression algorithms.
1
P
a
0
0.1
E
a
5
.01
.-
m
K
.
c
0 00
100 1000 10
Array Size Elements
Figure 5 Clutter Amplitude is Decreased by
Reducing Sensor IFOV (90
x
90 FOV)
Track file filters are the simplest clu
discriminators. The successive locations of
potential threat object are associated, and
resulting track is subjected to a variety
reasonableness tests. The processing loads
modest because the track file system is a p
detection system. To limit the number of candid
objects, the threshold is normally adjusted usin
CFAR method. The most difficult problem for tra
file filters is the association problem of link
observations from consecutive frames. Associa
is normally performed using a temporal projec
that defines a window in the next frame that sho
contain the track file object. The accuracy of t
projection depends directly on the sensor resolu
which favors a higher resolution sensor.
Velocity filters operate on the entire scene since th
are pre-detection filters. The ability to discrimin
velocity is the key
to
success in this method. Sinc
is relatively easy to determine the
LOS
rate for
inertial background (terrain), velocity filtering is m
effective in separating threats from terrain. T
limiting feature is the velocity resolution (Figure 6)
0 2 4 6 8 10 12 14 16 18
Range Km
Figure
6
Increased Resolution Increases the
Discrimination Range of Velocity Filtering
5.1 18
-
8/9/2019 First Electro Optical System of the 21st Century
5/8
-
8/9/2019 First Electro Optical System of the 21st Century
6/8
pixel, the baseline six sensor DAlRS system
operating at 30 Hz frame rate requires about 29-42
billion arithmetic operations per second. Operation at
60
Hz, will double the required operations.
Resolution
Sensitivity
Defensive IRST
The memory requirements are dominated by the
RE
algorithm. The input image is expanded
to
the high
resolution format using two gradient images and
other intermediate results. For 2:1 resolution
enhancement, the total memory storage is 17 pixels
per input pixel. Thus each 1024x1024 sensor
requires 17 megawords 34 megabytes) of storage.
If we can provide processing to achieve the 4:l
resolution enhancement that the optics can support,
the required storage grows
to
65 pixels per input
pixel
or
130 megabytes of storage per sensor.
7.5 BOPS
0.6 BOPS
1.25-15 GFLOPS 1-2 MByte
The IRST algorithms have a range of requirements
between 29 (optimistic) and 2,000 (very pessimistic)
operations per pixel A reasonable estimate is 50-100
operations per pixel. However, there is a very real
probability that the required frame rate can be
reduced
to
5-10 Hz by skipping frames without
significant impact on Pd or Pfa. Also, it is almost
certain that use of off-board data can also reduce
this number.
The resulting load will range from 1.5
to 18 GFLOPS for the IRST algorithms. These are
the pixel level algorithms and do not include the track
file, IMU interface and sensor management
algorithms. While those algorithms will represent the
largest effort in software development, they will have
negligible impact on the throughput requirements. A
typical estimate would be 1,000 CFAR candidates
per frame, each requiring 10,000 instructions per
candidate which yields a load
of
0.3 BIPS.
Missile threat warning, requires fewer than 10
operations per pixel. Additional operations may
prove beneficial in exploiting the high resolution of
the DAIRS, but this has not yet been established.
30
operations per pixel is the current upper bound
estimate, even with full clutter suppression
processing. Assuming the use of 2:l resolution
enhanced data, the required throughput is 7.5-22.5
BOPS at
30 Hz.
Considering the threat regions that
are not approachable by an attacking missile can
reduce
this
by about one half.
For the sky regions,
there is no benefit to using the resolution enhanced
data. This reduces the rate in that region by
4:l.
This more complex sensor management strategy
reduces the required throughput to 3-9 BOPS.
The resulting throughput and memory requirements
are summarized in the Table
I.
No margins are
applied. All entries are additive.
I DAIRS Processor Requirem
Throughput I Memo
Enhancements I 12 BOPS 170 Mbyte
NUC I 1.2-1.5BOPS
I
Totals 29-42 BOPS
I
175 MByte
Table
I
Summary Processing and Memory
Requirements for DAlRS
Alternate Applications of
DAlRS
In addition to the JSF role, Dairs has also be
studied for use on other platforms (Figure8).
Figure 8
-
DAlRS Candidate Platforms Span th
Range of Military Applications
Existing fixed and rotary wing aircraft can bene
from the multi-function capabilities
of
distribut
aperture systems as a path
to
increased survivabil
Rotary wing aircraft will benefit from high frame rat
due
to
the high scene motion rates for nap-of-th
earth operation.
For armored vehicles, the threat
top down attack requires that an early warni
capability be added to existing vehicles, together w
countermeasures that can prevent weapons fro
reaching the vulnerable top surfaces. In addition, t
ability
to
provide functions equivalent to the existi
commanders viewer, gunners weapon sight a
5.1
20
-
8/9/2019 First Electro Optical System of the 21st Century
7/8
driver's viewer while reducing signature and
eliminating gimbaled sensing are all attractive
features. Shipboard defense against anti-ship
missiles and threat aircraft has included vigorous
pursuit of passive E-0 sensors due to their high
spatial resolution compared to
RF
sensors. The
scanned sensor technology previously applied
proved to be heavy and expensive, and placed
claims on mast space that could serve other
functions more efficiently. DAIRS, with its small size
and absence of gimbal mechanisms is projected to
save 60-70percent of the weight and cost of a
scanned system while coexisting with other mast
mounted systems.
Future Trends in
E 0
System Design
Two features will dominate the design of
E-0
systems for the near future:
Increased unctionality, and
Large processing growth.
Application of the resolution enhancements being
developed for DAlRS will permit the design of
gimbaled sensors with simplified optics. The
multiple fields of view needed to optimize search,
detection, recognition and weapon delivery will be
obtained from a single large optical field of view
using processing to achieve the higher resolutions
needed.
Gimbal elimination will remain an elusive goal, but
may be achieved using large aperture DAlRS
designs. This is still an attractive solution because
the gimbal mechanism is the most expensive part of
current
E - 0
sensors. The gimbal is currently
justified by the need for a laser illuminator for laser
guided weapons. If these are replaced by more
covert and self contained weapons, then the
elimination of the gimbal will be practical with
considerable benefit to airborne platforms.
The capabilities of the DAlRS sensors remains
limited by the size of detector arrays.
The small
market base for ever larger arrays will inhibit their
development, particularly if processing can be shown
to achieve the same result using resolution
enhancement techniques. The fundamental
physical limits of optics resolution will also be
overcome using advanced processing methods and
large stored data bases. Efforts in neural networks
can increase resolution. Extending the optical
resolution by even a factor of two is a substantial
benefit. Neural network resolution enhancement
can be performed in a context free mode, but, in the
near term, providing a context basis using stored
data will be an attractive approach requiring
substantially ess computation.
Sensitivity is also limited by detector physics. The
dynamic range of a photovoltaic detector is limited by
its charge storage capacity. The dynamic range
requirements for IR scenes are determined by the
scene equivalent temperature excursions and the
noise equivalent temperature difference (NETD) of
the sensor system. The environment is usually
taken to span an intrascene dynamic range of
80
100 K which results in occasional saturation in non-
combat conditions, and frequent saturation in
battlefield conditions. Current sensors have
achieved NETD near .01 K and efforts continue to
achieve NETD of .001
K.
The required dynamic
range is in the range of lo (14 bits / benign
environment) to lo6 (20 bits
/
battlefield
environment). Available arrays store 10 electrons
per detector which is less than
12
bits dynamic
range. (Dynamic range for high background IR
sensors is limited to the square root of the electron
storage capacity.) Experimental arrays with 2.5E7
stored electrons per detector have been fabricated.
Achieving a
20
bit dynamic range would require
increasing electron storage by
lo5
which is
problematic.
Raising the frame rate and using post readout
integration is a feasible alternative. The result
would be arrays with millions of detectors being
sampled at hundreds of frames per second. The
increased frame rate is attractive because it allows a
more dynamic environment with greater bandwidth
for platform and scene motions, while providing
adaptive response to scene signatures at the pixel
level. Sensor sensitivity and speed of response
becomes a software function which opens the design
trade space to developments that cannot be
foreseen today. Experimental
51
2x51
2
arrays
operating at 480
Hz
have been built. Multiport
arrays with data rates of
lo9
samples per second
with 16 bit dynamic range can be anticipated from
current developments.
Multi-spectral sensors are another area of active
research that will impact data processing
requirements. Current designs range from two
color sequential sensors to multi-spectral sensors
that provide data at a large number of wavelengths
simultaneously. Using existing staring detector
arrays imposes a spatial, temporal and spectral
trade-off in which the num ber of wavelengths, the
array spatial coverage and the time (number of
frames) needed to satisfy a required spatial
coverage are combined to satisfy a mission
5.1 21
-
8/9/2019 First Electro Optical System of the 21st Century
8/8
objective. Two difficulties are the registration of
time displaced spectral observations that is required
when all of the spectra are not available
simultaneously
for
the same spatial locations, and
the problem of radiometric conversion of detector
spectral data in the presence of readout errors. The
detector errors are similar to the NUC errors
discussed above, with the addition
of
detector
spectral gain errors. Difficulties in characterizing all
of the errors in a dynamic environment must be
overcome to enable spectral techniques. Methods
currently being developed to determine detector
nonlinear characteristics may be necessary, as well
as methods to establish temporal characteristics.
This could lead to multi-dimensional NUC tables that
would require significant increases in processing
power.
CONCLUSION
Distributed aperture sensor systems are a new
direction in electro-optics. Large staring focal plane
arrays support multiple functions and eliminate the
costly stabilization and pointing gimbals of current
systems. Initial designs are processing intensive
due to high data rates, increasing functions per data
sample and inherent characteristics of arrays with
one million detectors or more. Entry level
processing capabilities are in the range of one
hundred billion arithmetic operations per second with
memory storage of several hundred megabytes.
This provides missile warning,
IR
search and track,
pilotage, battle damage assessment and weapon
delivery support. Distributed aperture systems are
widely applicable to military systems and are likely to
be evolved for multiple platforms. Growth will occur
in both the size and speed of the detector arrays, but
also in the applications that the data will support.
Throughput increases of one
to
two orders of
magnitude will happen when the processing
becomes economically available. These
developments will use image understanding
technologies that are already in place. Continuing
evolution in that area will result
in
even greater
requirements or increased processing.
REFERENCE
[ l]
Hale,
R.
A. et al United States Patent No.
5,317,394 Distributed Aperture Imaging and
Tracking System issued May 31, 1994
[2] Hale,
R.
A. et al - United States Patent No.
5 4 12,421 Motion Compensated Sensor issued
May 2,1995
[3]
R.
C. Hardie and E. Kaltenbacher
-
High
Resolution Infrared Image Reconstruction using
Multiple, Low Resolution, Aliased Frames
-
to be
published
[4] ONeil
W .F,
Experimental Verification of Dithe
Scan Non-uniformity Correction, 1996 meeting o
the
IRIS
Specialty Group on Passive Sensors, March
13,1996
[5] Schaum, A. and McHugh, M., Analytic Methods
of Image Registration: Displacement Estimation and
Resampling, NRL Report 9298, Feb. 28,1991
[6] Hench, D. and Fried, D., Status o
Pseudoregistration Development BC-276, the
Optical Sciences Company, Placentia CA., Feb
1985
[7] Kuglin, C. and Hines, D., The Phase Correlatio
Image Alignment Method.
IEEE
Proc. 1975 Conf. on
Cybernetics and Society, Sept. 23-25, 1975 pp. 163
165
[8] McGlynn, J. D. and Sofianos, D.
J.,
Parametric
Model-based Characterization of
IR
Clutter, Science
Applications International Corporation Repor
(undated)
[9] J. M. Mooney et al, Responsivity Nonuniformit
Limited Performance of Infrared Staring Cameras
Optical Engineering, Nov. 1989,
Vol
28, NO. 11, p
1151
5.1 22