noise at the system levelkilopixel.net/publications/isscc 2007 - rlb v4.pdf · 2020. 10. 18. ·...

ISSCC 2007 Forum:Noise in Imaging Systems

February 15, 2007

Noise at the System LevelRick Baer

21/5/2007

Introduction

3

Micron Confidential

1/5/2007

The systems view

Noise sources

Noise modifiers

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance

color correct sharpen

noise reduce

gamma & tonedemosaic

compresscolor space output

4

Micron Confidential

1/5/2007

Major CCD and CMOS noise sources

• BothFlareVignettingPRNU (pixel lithography)Photon shot noiseRead noise (thermal, 1/F, kT/C)Row reference

• CCDColumn dark current variationSmearBlooming

• CMOSColumn gain variationPGA / ADC gain and offset variation

5

Micron Confidential

1/5/2007

Major noise modifiers

Increases noise:

• White balance

• Color correction

• Sharpening

• Tone mapping

Decreases noise:

• FPN correction

• Noise reduction filtering

• compression

Diffuses noise:

• Pixel reconstruction

• Color space conversion

61/5/2007

Overview

• There are many sources of noise and interference in imaging systems

• Image processing has a big effect on noise at the systems level

• Noise measurement is complicated by the (unknown) effects of image processing

71/5/2007

Noise from the perspective of image processing

81/5/2007

Pixel / circuit noise sources(reviewed in other ISSCC noise forum presentations)

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



91/5/2007

Exposure control

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



10

Micron Confidential

1/5/2007

Simplified noise / exposure control model

X ADC

Texp Analog gain

Dark currentnoise

+ +Xsignal

Read noise (N1)

+

ADC input, quantization noise (N2)

Exposure controls

11

Micron Confidential

1/5/2007

SNR variation

2

2N22

N1 G/S σσ +=S/N

Gain

S/N

mi ma

The signal to noise ratio increases with gain until it is limited by noise source N1.

12

Micron Confidential

1/5/2007

DR variation

2N2

2N1

2G/552 σσ +=k

DR

Gain

DR

mi ma

The dynamic range decreases with gain.

13

Micron Confidential

1/5/2007

Exposure

Short exposure(noise, quantization)

Long exposure(clipping)

Ideal exposure

14

Micron Confidential

1/5/2007

Flicker suppression

desk lamp

0.6000

0.8000

1.0000

1.2000

1.4000

1.6000

1.8000

0 2 4 6 8 10 12 14 16 18

time [msec]

rela

tive

inte

nsity

60W tungfluor ring

Restrict exposure and frame periods to multiples of the flicker period (e.g. 1/120 Hz)

15

Micron Confidential

1/5/2007

What’s wrong with this picture?

16

Micron Confidential

1/5/2007

Importance of exposure control

• Underexposure leads to poor SNR and possibly posterization

• Underexposure may lead to poor color reproduction (due to pixel non-linearity)

• Overexposure leads to clipping

• Incorrect exposure timing may cause flicker to appear

171/5/2007

Offset / gain correction

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



18

Micron Confidential

1/5/2007

Sources of offset FPN

• Row FPN Optical black clamp circuit (CCD & CMOS)Power supply noise (CMOS)

• Column FPNDark current (CCD)

Column offset (CMOS)

• Random FPNDark current (CCD & CMOS)

19

Micron Confidential

1/5/2007

CCD and CMOS readout timing

row 1

row 2

row n

row 1

row 2

row nreset

integrate

Transfer (readout)digitize

time

CDS row 1row 2

row n

CCD

“SCDS”4T RS CMOS

3T RS CMOS

Temporal separation introduces susceptibility to dark current, power supply noise

20

Micron Confidential

1/5/2007

Visibility of row/column FPN

random / row ratio

21 5 10

Row noise is still visible, even when it is buried in 5x random noise

21

Micron Confidential

1/5/2007

CCD optical black reference pixels

leadinghorizontal

OB trailinghorizontal

OB

leadingvertical

OB

trailingvertical

OB

Sony ICX085

22

Micron Confidential

1/5/2007

Horizontal optical black signal vs. row number

average of 40 columns

23

Micron Confidential

1/5/2007

CCD row FPN offset correction (OB clamp)

OB accumulator

+−

+

Pixel data

Pixel ID

Methods:1. Accumulate leading/trailing OB pixels (analog / digital)2. Multi-row running average (digital)3. Exclude outlying pixels (digital)

24

Micron Confidential

1/5/2007

CCD column FPN (vertical register dark current)

Generally buried in read noise: visible at high temperatures

25

Micron Confidential

1/5/2007

CMOS row FPN offset correction

S/H

S/H

Σ

+

S/H

Σ

+

S/H

Σ

+

S/H

Σ

+

average

- - - -

Pixel circuit

Column bufferOptical black

26

Micron Confidential

1/5/2007

CMOS row offset features

• Analog or digital offset subtraction• Provides power supply noise rejection (for 3T CMOS)

• Dark current clamping isn’t required

• Dark current can introduce row noise!

27

Micron Confidential

1/5/2007

Sources of gain FPN

• PRNU (random)• Column gain variation (column)

• ADC/PGA gain variation (block / color plane)

28

Micron Confidential

1/5/2007

Gain FPN (multiplicative noise)

Column gain

ADC/PGA gain (four blocks)

PRNU

Assign each color plane to a separate ADC/PGA to avoid visible gain errors.

29

Micron Confidential

1/5/2007

Gain FPN correction

Inverse flat-field image

XData in

address

Data out

1/(column gain)

XData in

address

Data outData in AWB

Data out

PRNU

column

PGA/ADC

Digital correction and memory is required!

30

Micron Confidential

1/5/2007

Effect of offset / gain correction

• FPN that would otherwise be intolerable may be almost completely eliminated

• Row FPN correction circuits may introduce their own FPN

• Spatially random offset / gain correction (dark current / PRNU) requires large memories (and large digital multipliers)

• Offset correction may suppress power supply noise

311/5/2007

Defect correction

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



32

Micron Confidential

1/5/2007

Defect correction

• Static defect tables• Dark current – works best on extraordinary pixels,

not on distribution tail

• On-the-fly correctionThreshold problems (video mode)

• Effect of correction (interpolation -> spatial correlation)

33

Micron Confidential

1/5/2007

Defect correction

Defect location

table

bypass / interpolate

Data in

addressaddress

Data out

Row buffers interpolator

exception detectorData in

Data out

Static defect correction

Dynamic defect correction

Row buffers interpolator

34

Micron Confidential

1/5/2007

Defect characteristics

White defects• Caused by dark current “tail”, impurities, dislocations• Temperature dependent• Generally single pixels• Correction works best on isolated “hot” pixels

Black defects• Caused by occlusions• Generally clusters of pixels

Dark current

Coun

t

Dark current

Coun

t

?

35

Micron Confidential

1/5/2007

Effects of defects / defect correction

• Defective pixels contribute additive or multiplicative noise

• Defect correction replaces bad pixels with interpolates of neighboring values, results in resolution loss.

• Resolution loss isn’t visible unless the number of defective pixels is large (> 0.1%).

• On-the-fly defect correction may introduce temporal noise (blinking pixels)

361/5/2007

Flare Compensation (ABL)

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



37

Micron Confidential

1/5/2007

Sources of flare

Reflections from:•Lens surfaces•Lens barrel•Aperature•IR filter•Cover glass•Sensor•Bond wires and pads

38

Micron Confidential

1/5/2007

Ghost images

39

Micron Confidential

1/5/2007

Non-uniform flare

40

Micron Confidential

1/5/2007

Reference column leakage

41

Micron Confidential

1/5/2007

Uniform flare compensation

10% flare

0% flare

-10% flare

42

Micron Confidential

1/5/2007

Effects of flare

• Uniform flare adds signal-dependent offset (noise)• Uniform flare can be subtracted. Over-subtraction

reduces color accuracy.

• Specular sources may produce ghost images which can’t be removed.

• Black row effects may appear because of light leaking into row reference pixels.

431/5/2007

Shading correction

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



44

Micron Confidential

1/5/2007

Vignetting and color shading

Lens vignetting: color planes share a common

center

Color shading: color planes have different centers (because

of pixel asymmetry)

Sources of shading•Lens vignetting•IR filter CRA dependence•Pixel angular response / asymmetry

45

Micron Confidential

1/5/2007

Shading correction

Uncorrected image

Corrected image

X

X

X

X

shading correction parameters

(from camera characterization)

Shading model

Approximate inverse flat-

field

Approximate inverse flat-

field

46

Micron Confidential

1/5/2007

Illuminant-dependent color shading

Shading correction optimized for CWF

D65 - daylight CWF - fluorescent “A” - tungsten

(color saturation increased to enhance visibility)

47

Micron Confidential

1/5/2007

Effects of vignetting / pixel angular response

• Reduced signal / SNR in corners• Color shading

• Gr/Gb channel imbalance

481/5/2007

Pixel reconstruction

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



49

Micron Confidential

1/5/2007

Pixel reconstruction

• Undersampling results in artifacts• Demosaic uses interpolation -> results in spatial

correlation of noise.

50

Micron Confidential

1/5/2007

Color artifacts caused by undersampling

51

Micron Confidential

1/5/2007

Noise correlation

G G G

G

GGG

G G

B B

B B

B BG G

GGR R R

R R R

Noise diffuses to adjacent pixels

52

Micron Confidential

1/5/2007

Effects of pixel reconstruction

• Reconstruction may introduce color artifact noise• Reconstruction diffuses noise to adjacent pixels

531/5/2007

White balance

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



54

Micron Confidential

1/5/2007

Effect of color temperature on RGB balance

red

green

blue

elec

tron

s/lu

x

Color temperature [K]

Sony CCD QE curves and CM-500 IR filter

55

Micron Confidential

1/5/2007

White balance operation

X

X

X

Σ

red stats

Σ Σ

green stats

blue stats

R

G

B

Scene rule base (e.g.

gray world)

56

Micron Confidential

1/5/2007

Effect of white balance

• Neutral colors balance to gray• Amplification of weak color channels increases noise

• Relative color strength depends on spectral response, illuminant color

571/5/2007

Color correction

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



58

Micron Confidential

1/5/2007

Color Processing Decreases the SNR

CA ⋅=BGR A transformation matrix is required to convert from the sensor color

space to the output color space

Consider a two-color world:

BRCBRC

2

1

+=+=

αα

Transform to color: Transform to monochrome:

212

221

1CCB

1CCR

αα

αα

−−

=−−

= ( )2121 CCX +=

( )22

221

21

NNSSNS R

αα

+

−= ( ) 2

221

21

NNSS

NS X

+

+=

• Color correction decreases the SNR• The greater the crosstalk, the greater the noise amplification

α = pixel crosstalk factor

sensoroutput

59

Micron Confidential

1/5/2007

Effect of color correction

• Color fidelity is improved

• Noise is significantly amplified by off-diagonal matrix terms

• Off-diagonal matrix terms depend on spectral response and pixel-to-pixel crosstalk

601/5/2007

Sharpening

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



61

Micron Confidential

1/5/2007

Sharpening filters amplify noise

original sharpen

original + noise

62

Micron Confidential

1/5/2007

Effect of sharpening

• The noise has more high-frequency spectral content than the image

• Sharpening filters amplify high-frequencies and therefore amplify noise

631/5/2007

Noise reduction filtering

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



64

Micron Confidential

1/5/2007

Noise filtering(topic of Dr. Pizurica’s ISSCC presentation)

• Temporal filtering• Spatial filtering

• Adaptive spatial filtering

65

Micron Confidential

1/5/2007

What distinguishes signal from noise?

• The signal is stationary (… but so is FPN)• The spatial frequency response of the signal is

limited by the system MTF

• The color gamut of the signal is limited by the surfaces / illuminants observed in the real world

66

Micron Confidential

1/5/2007

Effect of noise reduction

• Noise is presumably reduced• Sharpness may be lost

• Texture detail may be lost

671/5/2007

Gamma correction and tone mapping

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



68

Micron Confidential

1/5/2007

Gamma correction and tone

• Gamma correction – compensates for display device• Global tone correction – produces pleasing tone

reproduction

• Local tone correction (e.g. Retinex) – for high DNR images

69

Micron Confidential

1/5/2007

Effect of tone correction on SNR

input

outp

ut

Gamma / tone function

Noise amplification

= slope of tangent line

signal amplification

= slope of line from origin

original

after gamma correction

The tone correction function must be reversed to make meaningful

noise measurements

701/5/2007

Color space conversion

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



71

Micron Confidential

1/5/2007

Color space conversion

R G B R G B Y u Y v Y u Y u

2 pixels 2 pixels

conversion

72

Micron Confidential

1/5/2007

Chrominance filtering

LPF

luminance

chrominance

73

Micron Confidential

1/5/2007

Effects of color space conversion

• Chrominance noise may be reduced• Noise may be spatially diffused

741/5/2007

Compression

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



75

Micron Confidential

1/5/2007

Compression

•Noise reduced•Severe block artifacts

•Noise significantly reduced•Loss of texture

JPEG 2000compression

JPEG compression

27x 27x

76

Micron Confidential

1/5/2007

Effects of compression

• Noise is reduced (similar to noise reduction processing)

• New sources of noise (block artifacts, quantization) may be added

771/5/2007

Measurement

78

Micron Confidential

1/5/2007

Measurement challenges

• Interpretation of luminance versus chrominance noise

• Dependence on viewing conditions (Johnson & Fairchile EI2004)

High frequency chromatic noise doesn’t effect quality• Nonlinear processing (un-map gamma)• Quantization limits on measurement• Noise decomposition (temporal, fixed, etc.)• (other practical stuff like shading suppression)• Compression / adaptive noise filtering

79

Micron Confidential

1/5/2007

ISO 14524 OECF measurement

Diffusing screen

Extended source

OECF test chart

System under test

Required to un-map gamma & tone correction functions

80

Micron Confidential

1/5/2007

ISO 9358:1994 veiling glare measurement

• Integral method (suitable for uniformly radiant scenes)

• Analytical Method (suitable for intense isolated sources)

Diffusing screen

Extended source

Black area

System under test

System under test

source

collimator

81

Micron Confidential

1/5/2007

ISO 15739 noise measurement

Diffusing screen

Extended source

OECF test chart

System under test

OECF measurement

patches

Noise measurement

patches

High frequency

measurement patches

82

Micron Confidential

1/5/2007

ISO 15739 noise component analysis (1)

++ -

FPN image

Temporal noise

images

(+ σ temporal)

>= 8 images

83

Micron Confidential

1/5/2007

ISO 15739 noise component analysis (2)

+++-

Spatially-random noise

image

FPN or temporal

noise image-

Row noise vector

Column noise vector

(+ σ random)

(+ σ random)

84

Micron Confidential

1/5/2007

Utility of noise decomposition

85

Micron Confidential

1/5/2007

Quantization limits

Noise below ½ LSB can’t be measured

861/5/2007

Conclusions

1/5/2007 87

871/5/2007

Image processing modifies noise …

pixel array

analog / ADC

offset FPN

correct

gain FPN

correct

defect correct

flare correct

shading correct

Exposure control

scene intelligence

white balance


noise reduce



Cancels / reduces noise

Generally increases noiseIncreases noise

1/5/2007 88

881/5/2007

… and makes it difficult to measure!

• Opto-electronic conversion function• Luminance vs. chrominance

• Adaptive processing

• Compression artifacts

The ISO 12232 digital photography speed measurement standard has not been widely adopted because of the difficulty of noise measurement.

1/5/2007 89

891/5/2007

Noise is one dimension of image quality …

Noise

Texture detail

Color saturation Resolution

Tone reproduction

Artifacts

1/5/2007 90

901/5/2007

… but they all matter!

Noise

Texture detail

Color saturation Resolution

Tone reproduction

Artifacts

91

Micron Confidential

1/5/2007

92

Micron Confidential

1/5/2007

References

• J. Nakamura, Image Sensors and Signal Processing for Digital Still Cameras, CRC Press, New York, (2005).

• S. Battiato, M. Mancuso, “An Introduction to the Digital Still Camera Technology”, ST Journal of System Research - Special Issue on Image Processing for Digital Still Camera, Vol. 2, No.2, (12/2001).

• G. Williams Jr., Ed., Digital Solid State Cameras: Designs and Applications, Proc. SPIE, Vol. 3302, (1998).

• N. Sampat, J. DiCarlo, R. Motta, Eds., Digital Photography, Proc. SPIE, Vol. 5678, (2005).

• N. Sampat, J. DiCarlo, R. Martin, Eds., Digital Photography II, Proc. SPIE, Vol. 5678, (2006).

• R. Martin, J. DiCarlo, N. Sampat, Eds., Digital Photography III, Proc. SPIE, Vol. 5678, (2007).

• J. Lopez-Alonso, R. Gonzalez-Moreno, J. Alda, “Noise in imaging systems: fixed pattern noise, electronic, and interference noise”, Proc. SPIE, Vol. 5468, p. 399-407, (5/2004).

• ISO 9358 – Optics and optical instruments – Veiling glare of image-forming systems – Definitions and method for measurement.

• ISO 12232 – Photography – Digital still cameras – Determination of exposure index, ISO speed ratings, standard output sensitivity, and recommended exposure index.

• ISO 14524 – Photography – Electronic still-picture cameras – Methods for measuring opto-electronic conversion functions (OECFs).

• ISO 15739 – Photography – Electronic still-picture imaging – Noise measurements.

Noise at the System LevelIntroductionThe systems viewMajor CCD and CMOS noise sourcesMajor noise modifiersOverviewPixel / circuit noise sourcesExposure controlSimplified noise / exposure control modelSNR variationDR variationExposureFlicker suppressionWhat’s wrong with this picture?Importance of exposure controlOffset / gain correctionSources of offset FPNCCD and CMOS readout timingVisibility of row/column FPNCCD optical black reference pixelsHorizontal optical black signal vs. row numberCCD row FPN offset correction (OB clamp)CCD column FPN (vertical register dark current)CMOS row FPN offset correctionCMOS row offset featuresSources of gain FPNGain FPN (multiplicative noise)Gain FPN correctionEffect of offset / gain correctionDefect correctionDefect correctionDefect correctionDefect characteristicsEffects of defects / defect correctionFlare Compensation (ABL)Sources of flareGhost imagesNon-uniform flareReference column leakageUniform flare compensationEffects of flareShading correctionVignetting and color shadingShading correctionIlluminant-dependent color shadingEffects of vignetting / pixel angular responsePixel reconstructionPixel reconstructionColor artifacts caused by undersamplingNoise correlationEffects of pixel reconstructionWhite balanceEffect of color temperature on RGB balanceWhite balance operationEffect of white balanceColor correctionColor Processing Decreases the SNREffect of color correctionSharpeningSharpening filters amplify noiseEffect of sharpeningNoise reduction filteringNoise filtering�(topic of Dr. Pizurica’s ISSCC presentation)What distinguishes signal from noise?Effect of noise reductionGamma correction and tone mappingGamma correction and toneEffect of tone correction on SNRColor space conversionColor space conversionChrominance filteringEffects of color space conversionCompressionCompressionEffects of compressionMeasurement challengesISO 14524 OECF measurementISO 9358:1994 veiling glare measurementISO 15739 noise measurementISO 15739 noise component analysis (1)ISO 15739 noise component analysis (2)Utility of noise decompositionQuantization limitsImage processing modifies noise …… and makes it difficult to measure!Noise is one dimension of image quality …… but they all matter!References

noise at the system levelkilopixel.net/publications/isscc 2007 - rlb v4.pdf · 2020. 10. 18. ·...

Documents