Image Based Prediction ofThermal Imaging

PerformanceSaar Bobrov

Yoav Y. Schechner

Department of Electrical Engineering,Technion

Acknowledgement: Rafael Ltd.

The Problem

• Predict sensor performance – Prior to operation– In specific scenes

Bobrov & Schechner

Applications• Space probes

• Electro-optical missiles

• Medical Probes

Bobrov & Schechner

Previous work• Performance based on imager

characteristics– Ratches 75’; Wittenstein and Gal 03’

• Synthetic scenes– Wegner and Drake 00’; Toler and Grey 80’

• Image based/Example based:– Texture synthesis

• Wei and Levoy 00’; Heeger and Bergen 95’– Image analogies

• Hertzman and Jacobs 01’; Drori and Cohen-or 03’

Systems Introduction

Hi-q

Lo-q

Hi-q imaging

Scene radiance

Lo-q imaging

High quality Thermal Sight

Low quality Missile seeker

Simulation

Imaging parameters

Bobrov & Schechner

From Photons to Gray Levels

PhotonRadiance

Optical Assembly

Scene

PhotonIrradiance

Detector Module

ElectronicSignal

Signal Processing

Gray levelImage

),(out nmI),( nmV

detE][

sec

scn

2

),(

srmcmphoton

yxL

),(),(),( scnoptdetSPscnout yxLyxLnmI TTTΤ

Bobrov & Schechner

Optical Assembly

Cold filter

Scanningmirror

Cold shield

Lens

Body

Bobrov & Schechner

Atmospheric Transmittance

Cold Shield

Cold shield Internal radiation

Internal radiation

Detector

Body

Internal radiation

Filter

Bobrov & Schechner

Optical Assembly

Cold filter

Scanningmirror

Cold shield

Lens

Body

Bobrov & Schechner

Optical Functional Diagram

PhotonRadiance

),(scn yxL

ProjectionProjectedradiance

),(proj yxE

BlurBlurred

radiance

),(blur yxE

][sec

scnoptproj

optblur

optrad

scnoptdet2

mcm

photonLLE ΤΤΤΤ

IrradiancePhoton

irradiance

),(det yxE

Bobrov & Schechner

Projection

Detector

Sceneradiance

optfR

2opt

scn R

A

][sec

scnproj2

optopt2

opt2

opt

2 ),(),(

mcm

photonfR

fR

R

A

fR yxLyxE

Bobrov & Schechner

Blur

projprojoptblur

blur ),( EhEyxE opt Τ

Optical PSF

Bobrov & Schechner

Opticsradiance

Direct ReflectedInternal radiance

Cold Shield

Sceneradiance

Detector

Camera body

Irradiance

rflCSHoptblurblurirradblur

det ),( EEEEEyxE Τ

Bobrov & Schechner

From Photons to Gray Levels

PhotonRadiance

Optical Assembly

Scene

PhotonIrradiance

Detector Module

ElectronicSignal

Signal Processing

Gray levelImage

),(out nmI),( nmV

detE][

sec

scn

2

),(

srmcmphoton

yxL

),(),(),( scnoptdetSPscnout yxLyxLnmI TTTΤ

Bobrov & Schechner

Detector Circuit

IntegrationCapacitor

InSb

Integrationswitch ReadoutOutput

Photons

Bobrov & Schechner

Detector Flow Diagram

Photon irradiance

Photo-detection

ElectronFlux

),(gene yxN

CrosstalkElectron

Flux

][detdetconvert

detxtk

detsamp

detnoise

detread

detdet),( voltEEnmV ΤΤΤΤΤΤ

Sampling Electrons

),(sampe nmN

),(det yxE),(e yxN

Noise

Readout

Electricsignal

),( nmV

),(e nmN

Bobrov & Schechner

Photo-Detection

][secd

darkdetgene 2d)(),(),(

cme

Q qAiyxEyxN

InSb

Photons

ElectronFlux

Bobrov & Schechner

Detector Flow Diagram

Photon irradiance

Photo-detection

ElectronFlux

),(gene yxN

CrosstalkElectron

Flux

][detdetconvert

detxtk

detsamp

detnoise

detread

detdet),( voltEEnmV ΤΤΤΤΤΤ

Sampling Electrons

),(sampe nmN

),(det yxE),(e yxN

Noise

Readout

Electricsignal

),( nmV

),(e nmN

Bobrov & Schechner

Crosstalk

sec

gene

gene

detxtke 2),(

cme

xtk NhNyxN Τ

Crosstalk PSF

Photons

Bobrov & Schechner

Detector Flow Diagram

Photon irradiance

Photo-detection

ElectronFlux

),(gene yxN

CrosstalkElectron

Flux

][detdetconvert

detxtk

detsamp

detnoise

detread

detdet),( voltEEnmV ΤΤΤΤΤΤ

Sampling Electrons

),(sampe nmN

),(det yxE),(e yxN

Noise

Readout

Electricsignal

),( nmV

),(e nmN

Bobrov & Schechner

Sampling

eyxyxNtyxNnmNdA dd),(),(),( einte

detsamp

sampe

Τ

α Δxd

β

Δyd

Bobrov & Schechner

Detector Flow Diagram

Photon irradiance

Photo-detection

ElectronFlux

),(gene yxN

CrosstalkElectron

Flux

][detdetconvert

detxtk

detsamp

detnoise

detread

detdet),( voltEEnmV ΤΤΤΤΤΤ

Sampling Electrons

),(sampe nmN

),(det yxE),(e yxN

Noise

Readout

Electricsignal

),( nmV

),(e nmN

Bobrov & Schechner

Noise• Noise components:

– Shot noise:– Residual Non-Uniformity

sampeshot N

RNU

Bobrov & Schechner

Lo-q Non-Uniformity

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

220

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

220

Bobrov & Schechner

Hi-q Non-Uniformity

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

220

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

220

Bobrov & Schechner

Noise• Noise components:

– Shot noise:– Residual non-uniformity– Excess noise

• RMS noise amplitude:

• Noisy image:

sampeshot N

2excess

2RNU

2shotnoise

),(),(),(),( noisesampe

sampe

detnoisee nmNnmNnmNnmN Τ

RNU

excess

Bobrov & Schechner

Detector Flow Diagram

Photon irradiance

Photo-detection

ElectronFlux

),(gene yxN

CrosstalkElectron

Flux

][detdetconvert

detxtk

detsamp

detnoise

detread

detdet),( voltEEnmV ΤΤΤΤΤΤ

Sampling Electrons

),(sampe nmN

),(det yxE),(e yxN

Noise

Readout

Electricsignal

),( nmV

),(e nmN

Bobrov & Schechner

From Photons to Gray Levels

PhotonRadiance

Optical Assembly

Scene

PhotonIrradiance

Detector Module

ElectronicSignal

Signal Processing

Gray levelImage

),(out nmI),( nmV

detE][

sec

scn

2

),(

srmcmphoton

yxL

),(),(),( scnoptdetSPscnout yxLyxLnmI TTTΤ

Bobrov & Schechner

Signal Processing

][),(),(),( SPmed

SPADC

SPadd

SP levelgraynmVnmVnmIout ΤΤΤΤ

Mediansubtraction

Electricsignal

),( nmV ),(med nmV

A/D Add

),(out nmI),(8bit nmI

Gray levelimage

Bobrov & Schechner

Analog to Digital Conversion

V (m,n) [volt]Vmin Vma

x

I 8bi

t(m,n

)

[Dig

ital

un

its]

255

Bobrov & Schechner

Signal Processing

][),(),(),( SPmed

SPADC

SPadd

SP levelgraynmVnmVnmIout ΤΤΤΤ

Mediansubtraction

Electricsignal

),( nmV ),(med nmV

A/D Add

),(out nmI),(8bit nmI

Gray levelimage

Bobrov & Schechner

Median Subtraction• Omits large DC signal before A/D• Subtracts non-uniformity between

detector elements

-10

10

30

50

70

90

0 10 20 30

-10

10

30

50

70

90

0 10 20 30

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

220

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

220

50 100 150 200 250 300

20

40

60

80

100

120

140

160

180

200

220

Bobrov & Schechner

Signal Processing

][),(),(),( SPmed

SPADC

SPadd

SP levelgraynmVnmVnmIout ΤΤΤΤ

Mediansubtraction

Electricsignal

),( nmV ),(med nmV

A/D Add

),(out nmI),(8bit nmI

Gray levelimage

Bobrov & Schechner

Line Add to Standard Format

DetectorArray

120 lines 240 lines

Bobrov & Schechner

Signal Processing

][),(),(),( SPmed

SPADC

SPadd

SP levelgraynmVnmVnmIout ΤΤΤΤ

Mediansubtraction

Electricsignal

),( nmV ),(med nmV

A/D Add

),(out nmI),(8bit nmI

Gray levelimage

Bobrov & Schechner

Problem•How does the scene look like??

From Gray Levels to Photons

PhotonRadiance

Optical Assembly

PhotonIrradiance

Detector Module

ElectronicSignal

Signal Processing

Grey levelImage

),(q-Hiout nmI),(q-Hi nmV

),(q-Hidet nmE

),(scn nmL

),(),(),( q-Hiout

1SP1det1optq-Hiout

1q-Hiscn )()()()( nmnmnmL ITTTIΤ

Hi-qimage

Hi-qimage

Bobrov & Schechner

Inversion Pitfalls•Deblurring

…Hi-q

Image Hi-qDeblur …

SimulatedScene …

Lo-qBlur …

Lo-qImage

Bobrov & Schechner

Blur Frequency Response

HHi-q

HLo-q

Bobrov & Schechner

Deblur Operation

1/HHi-q

HLo-q

HLo-q/HHi-q

Bobrov & Schechner

Noise Handling• Hi-q image is noisy:

• No inversion to noise• Adding Lo-q noise

• Estimate Hi-q noise• Calculate Lo-q noise amplitude:

2q-Hinoise

2q-Lonoise

q-Lonoise )()(ˆ

noiseout

idealout

q-Hiout III

q-Hinoisenoise

sampee NNNN

q-Hinoise

Bobrov & Schechner

Experiments

Courtesy of Rafael Ltd.

Hi-q Image

Bobrov & Schechner

Lo-q Image

Bobrov & Schechner

Simulated Image

Bobrov & Schechner

Hi-qLo-q

Simulation

Bobrov & Schechner

Hi-q Image

Bobrov & Schechner

Lo-q Image

Bobrov & Schechner

Simulated Image

Bobrov & Schechner

Hi-qLo-q

Simulation

Bobrov & Schechner