digital camera and scanner performance evaluation: standards...

1

Digital Camera and Scanner Performance Evaluation: Standards and Software

Peter D. Burns & Don Williams

Updated course on imaging performance measurement methods for digital image capture devices and systems.

• several ISO measurement protocols for camera resolution, tone-transfer, noise, etc.

• underlying sources of variability in system performance

• measurement for emerging standards

• how standards can be adapted to evaluate capture devices

• evaluation of capture devices from cell phone cameras to scientific detectors

• required elements of software tools

We will also address various challenges to reliable evaluation and system comparison using actual test data and software tools.

SPIE/IS&T Electronic Imaging Symposium, 2014Course SC807

2 Feb. 2014, 8:30 AM-12:30 PM

Digital Camera and Scanner Performance Evaluation: Standards and Software,

Peter Burns & Don Williams

OBJECTIVES

This course will enable you to:

• Understand the difference between imaging performance and image quality

• Interpret and apply the different flavors of each ISO performance method

• Identify sources of system variability, and understand resulting measurement error

• Distill information-rich ISO metrics into single measures for quality assurance

• Adapt standard methods for use in factory testing

• Select software elements (with Matlab examples) for performance evaluation programs

• Be aware of upcoming standard measurement protocols

Burns/Williams, EI2014 2

Course SC807

2


Course Outline

1. Introduction– Measurement and sources of variability

2. Imaging Performance Measurement– Signal

• Opto-Electronic Conversion Function (OECF) • White Balance

– Color Evaluation• SFR/MTF & Summary measure: • Resolution, Acutance, Sampling Efficiency

– Image Noise (10 pages)• Random, fixed pattern, two-dimensional, streaks• Standard noise performance measurements and testing

Break3. Why results vary and ways to control variation

– Test plans– Requirements for evaluation tools

4. Standards Rodeo

4

What makes a digital image ?you push the button and …..

Burns/Williams, EI2014: Introduction

imageformation

capture processing displayhuman-to-machine

interface

intended renderingsharpeningde-noisingcompression

detector sizeshutteringreadout

renderingsize

environmentshutter lagcamera motion

opticsfocus

Object/scene

lightsource

Imaging Performance Metrics quantify how an imaging system or component acts on, modifies, or limits the effective optical characteristics of an input scene.

Once digitally captured, the image is encoded as N-bit (M channel) digital files

3


Digital Capture:Why measure imaging performance?

Standard testing, e.g., ISO– Aimed at hardware evaluation or benchmarking– Identify the influence of image processing, e.g., sharpening– Marketing leverage– Quality control and industry compliance

System testing– Actual product usage– Robustness testing – image processing as used, or in default signal

path, e.g., sRGB metric – Scanner as instrument

Metadata population– Enablement of enhanced product features and ease of use

Image Quality – Required as input for image quality


6

A shopping list of digital camera imaging (capture) performance features *

__________________________* From : “Proposal for a Standard to Test Mobile Phone Cameras”

Dietmar Wueller, Image Engineering, http://digitalkamera.image-engineering.de/

High Priority• OECF – tone reproduction• White balancing • Dynamic range (related scene contrast) • Used digital values • Noise, signal to noise ratio • Resolution (limiting resolution center, corner) • Sharpness

Recommended• Spatial distortion • Shading / vignetting• Chromatic aberration • Color reproduction • Shutter lag • Aliasing artifacts • Compression rates • Exposure and exposure time accuracy

and constancy • ISO speed

Optional• Detailed macro mode testing (shortest shooting distance, max. scale, distortion)

• Flash capabilities (uniformity• Image frequency • Video capabilities

(pixel count, resolution, frame rate, low light behavior)

May be tested if available and applicable. • Optical stabilization • Auto focus accuracy and constancy • Metadata (Exif, IPTC) • Watermarking • Spectral sensitivities • Bit depth of raw data • Detailed noise analysis • Color resolution (quantization)


4

7

How is imaging performance measured?

• Signal

Any response that provides valued information

• Noise

Any response that detracts from a desired signal

• Signal-to-Noise ratio

A good-to-bad proportionality measure


8

Vocabulary – Image Speak- staying dry in a storm of idioms -

Resolution

Exposure

Sharpening

Delta E

White Balance

Dynamic Range

GammaExposure

Depth offield

Flare

Shading

Noise

Geometric Distortion

Wobble

Aliasing

Signal Noise

Lateral ColorError Sharpness

Acutance


5

9

Imaging Performance vs. Image Quality

Image Quality: Task dependent, but can almost always be correlated to some weighted combination of imaging performance metrics.

• Aerial Reconnaissance – high SFR and low noise• Health Imaging – tone control, low noise, SFR dependent on task• Consumer Imaging – color and tone, moderate SFR.

Imaging performance specifies the underlying design needed to deliver perceived image quality.

Performance Metrics

Signal

OECF – tone, colorSFR / MTF

Noise

Total RMS noiseStructured noise

Artifacts

Quality Metrics

Subjective Correlates

SharpnessGraininess

ColorfulnessOver sharpness

Objective Utility Correlates

NIIRSBriggs scores

Resolving powerOCR detection


Burns/Williams, EI2014: Imaging Performance Measurement 10

SIGNAL

• Opto-Electronic Conversion Function (OECF)

• Spatial Frequency Response (SFR)

6


Signal – Metrics we’ll discuss

OECF

- Speed, Exposure

- White Balance

- Color Encoding

Spatial Frequency Response (SFR)

- Sampling Frequency

- Resolution, Sampling Efficiency

- Sharpening, Acutance

Burns/Williams, EI2014: Imaging Performance Measurement


Large Area Image Capture – OECF*

• Large area image capture behavior is measured by OECF• The conversion relationship between optical reflectances (gray levels) of a

source object and corresponding electronic (digital) count values in a digitized image file; a luminance-to-signal mapping relation.

Uniformstepsused

Input Signal ( stimulus)

Out

put

Sig

nal

(res

pons

e)

*Electro-optical Conversion Function

Burns/Williams, EI2014 Imaging Performance Measurement

7

13

0

50

100

150

200

250

0.000 0.200 0.400 0.600 0.800 1.000

neutral ( gray) reflectance

aver

age

gree

n co

unt v

alue

( 8

bit)

LightLow density

Darkhigh density

• Important link between the physical characteristics of the source object and its digital image file

• A prerequisite for facilitating image reproduction and image quality and color management tasks.

Some important items to keep in mind on OECF

*ISO 16067-1,16067-2, 21550, 12233, 15739, 14524

• Allows performance evaluation in a common “input referred” metric* for other metrics like resolution, dynamic range

• Measures signal ‘encoding’

• There is no good or bad OECF….but characteristics like clipping and strong curvatures with multiple inflection points should be avoided



Some common input/output OECF variables

Stimulus ( Input )

PhotonsReflectance

DensityExposure ( Lux-seconds)

Log exposureLuminance ( L*)

Response (Output)

Electrons, Current8 bit Count value (CV)

16 bit Count value (CV)Log CV


8

15

How OECF Measures Signal

Some ways signal is measured:

a) Peak-to-absolute zero – The differencebetween CV of interest and zero.

b) Peak-to-peak – Relative differencebetween the CV of interest and thelowest effective CV.

c) Incremental slope – The differencebetween the CV of interest and the CV ofan incrementally small change in density. The slope of the OECF curve for any given density. Indicates how well a device

can detect small density differences. (ISO)

OECF for Film Scanner

0

50

100

150

200

250

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

Density

CV

a) b)

c) slope


16

Camera SpeedISO 12232

2 – signal - speed

• Exposure Index (EI)

• Saturation-based speed (Ssat)

• Noise-based speed (Snoise)

Camera speed is intended to indicate the exposure range (or the lower exposure limit) over which a camera

can deliver a useful digital image.


9

17

Saturation-Based Speed, Ssat

Ssat= 78 / Hsatwhere

Hsat = Minimum focal plane exposure that yields the highest unclipped output signal( lux-seconds)

• Appropriate for controlled illumination environments where the best possible image quality is desired

• Intended to prevent clipping artifacts associated with image sensor saturation

2 – signal – speed

0

50

100

150

200

250

0.000

aver

age

gree

n co

unt v

alue

( 8

bit)

Exposure at detector

Correlates the highest exposure to the best SNR and the best image quality for a given camera. Does not mean that the same Ssat will yield equivalent image quality between cameras

• Currently what we think most camera manufacturers are using to report speed


18

Snoisex= 10 / Hs/nx

where Hs/nx= exposure which yields a camera

incremental SNR of x=40 ( excellent ) or x=10 ( acceptable ).

Based on an objective correlation tosubjective judgments of noise level acceptability.

Noise-based Speed

Currently, the best speed based path for comparing image quality equivalence across cameras


10

19

Color MetrologyJust a three channel OECF ?

• Keep the neutrals neutral. 85% of good color imaging is maintaining the neutrality of gray patches –white balance

• Color capture easier to manage for scanners because the device illumination is known.

.Burns/Williams, EI2014: Imaging Performance Measurement

20

Spatial Frequency Response (SFR)

NEXT


11

21

…and what are all these related terms?

• Number of pixels • Sampling frequency – dpi, ppi• Limiting resolution, Resolving power*• Spatial Frequency Response – SFR*• Modulation Transfer Function – MTF*

*Related Standards – ISO 16067-1, 16067-2, 12233, & 15529 all use these metrics.

2 – signal – resolution

Resolution and Spatial Frequency Response

What is it?The ability of an imaging component or system to distinguishfinely spaced detail : ResolutionGenerally, the ability to maintain the relative contrast of finely spaced detail. Spatial Frequency Response


22

In digital imaging two factors dictate the level of spatial resolution or detail that can ultimately be captured.

– Quantity: sampling frequency ( i.e. dots per inch, Mpixels )

• Sets the maximum resolution limit that can be achieved.

– Quality: SFR – Spatial Frequency Response

• Defines the real level of resolution that optics, environmental factors, hardware, and image processing impose on the captured image.

Resolution for Digital Capture Devices


12

23

Point Spread Function and image sampling

• Several imaging mechanisms influence spread function descriptions

– Optical : focus, diffraction, aberrations, radiation scatter, filters

– Mechanical : sampling aperture size, vibration, motion

– Digital filtering : de-screening, unsharp masking

• Some do not *

– Image noise, film grain, detector noise

– Sampling artifacts

– Signal quantization and clipping (thresholding)

*...but make spread function and SFR measurements difficult


24

Hubble Space Telescope Examplesame # pixels, different resolution

Before lens modification(oops !)

After lens modification(that’s better)


13

25

2 – signal – MTF

Pro ConDirect

Sine waves Simple, intuitive analysis Large spatial extentLimited sampling

Costly capture targets

Indirect

Point/Line Spread Fundamental features Low signal strength

Function

Square Waves Simple targets Large spatial extent

Edge gradients Compact features Noise sensitiveSimple targets

Noise fields Small signal Spectrum estimation

MTF/SFR Measurement Techniques


26

*Spatial Frequency Response

Spatial Frequency Response - SFR

Test Pattern

Incr

easi

ng s

patia

l fre

quen

cy

Lens

Imaged Test

Pattern

imagedlight

distribution

SFR or ( MTF)

Increasing spatial frequency

The relative change in contrast of increasingly finer spatial features


14

27

Theory: Frequency response by edge-gradient analysis

Theory: Estimate the line spread function from an ideal edge measurement

DerivativeFourier

transform

Line spreadfunction

Modulationfunction

Edgeresponse

Fouriertransform

Line spreadfunction

Modulationfunction

ISO 15529 Method: Principles of measurement of MTF of sampled imaging systems

Technical background


28

How is SFR measured for digital capture?

All you need is an image of a good slanted edge target.


15

29

y = mx + bProjection of data along the edge effectively increases sampling.

x

signal

ISO method: Edge profile


30

ISO Method: Resolution Charts

• OECF• SFR

Inputexposure

Outputsignal

2 – signal – targets and tools


16

31

Different SFR shapes and their meaning

Limiting resolution will generallycorrespond to the spatial frequency having a 0.10 SFR level

Increasing spatial frequency. Finer detail

0.0

0.5

1.0SF

R re

spon

se

Sharpened

Normal

Low contrast - Flare


32

How limiting resolution can fail to predict quality and why the SFR is better.

2 – signal – MTF


SFRs for ISO 12233 TargetSame Resolution, Different SFRs

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Frequency (cyc/pixel)

Sp

atia

l Fre

qu

ency

Res

pon

se

right image middle image left image

17

33

Derivative Measures of SFR

• Color fringing vs. misregistration

• Sharpening

• Acutance

• Aliasing

• Flavors of resolution ( 10% SFR, 50% SFR, half -Nyquist)

• Flare


34

Well-behaved SFRs

- Linear falloff to half-sampling frequency- Tightly grouped color SFRs- Similar form in both directions

Horizontal direction Vertical direction


18

Burns/Williams, EI2014: Imaging Performance MeasurementBurns/Williams, EI2014 35

Anatomy of the SFR- strange, but explainable, shapes -

Typical Optical SFR- focal plane referred -

0

0.2

0.4

0.6

0.8

1

1.2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

cycles/pixel

SF

R

- half sampling - 166 cy/mm for 3 micron sensor250 cy/mm for 2 micron sensor

Typical Sharpening Filter SFR(e.g. sparsely populated 1x5 FIR filter )

0

0.5

1

1.5

2

2.5

3

3.5

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

cycles/pixel

SF

RCombined delivered file SFR

Yes, these shapes are real and explainable !

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

cycles/pixel

SF

R

36

What’s the difference ?

• Resolution – the maximum spatial frequency of utility for an imaging system (limiting resolution).

• Sharpening- a class of image processing operations that enhances the contrast of selective spatial frequencies, usually visually important ones.

• Acutance – An integrated and weighted SFR ratio that is usually well correlated to sharpness.

• Sharpness – The visually perceived quality of being crisp or of containing detail.


19

37

Sharpness vs. Resolution

0.5

1.0

0.1

0.25 0.5

Frequency, cy/pixel

SF

R

high sharpness, low resolution

0

0.5

1.0

0.1

0.25 0.5

low sharpness, high resolution

0.5

1.0

0.1

0.25 0.5

Frequency, cy/pixel

SF

R


0

0.5

1.0

0.1

0.25 0.5


0.5

1.0

0.1

0.25 0.5

Frequency, cy/pixel

SF

R


0

0.5

1.0

0.1

0.25 0.5


0.5

1.0

0.1

0.25 0.5

Frequency, cy/pixel

SF

R


0

0.5

1.0

0.1

0.25 0.5



38

SFR and sampling efficiency

HVAC example

Advertised BTUs × Rated Efficiency = Usable BTUs

( 70K BTUs × 80% = 56K BTUs )the percentage of fuel that is converted to useable energy

Digital Camera example

# Delivered MPixels × Resolution Efficiency = # Effective MPixels( 6 MPixels × 80% = 4.8 Mpixels )

the percentage of optically resolvable pixels


20

39

1- Rayleigh Criteria - Find frequency of first 10% SFR occurrence in both vertical and horizontal directions. ( clip at 0.5 cycles/pixel)

2- Efficiency- Normalize each by digital limit of 0.5 cycles/pixel.3- Take product of vertical and horizontal components.4- Multiply by camera’s finished file size

6.0 MPixel camera SFRs

0

0.2

0.4

0.6

0.8

1

1.2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

theo

retic

al m

axim

um f

requ

ency

A

B

C

D

Cameras A&B:

6.0 MP x (0.5/0.5)2 = 6.0 MP[ E=100% ]

Cameras C&D:

6.0 MP x (0.3/0.5)2 = 6.0 MP[ E = 36% ]

Sampling efficiency for digital cameras



Some Popular Sharpening Options

21

41

..and their SFR responses

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.1 0.2 0.3 0.4 0.5 0.6

relative frequency

SF

R

reference SFR moderate strong

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.1 0.2 0.3 0.4 0.5 0.6

relative frequency

SF

R

reference SFR moderate strongreference SFR moderate strong

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.1 0.2 0.3 0.4 0.5 0.6

relative frequency

SF

R


0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.1 0.2 0.3 0.4 0.5 0.6

relative frequency

SF

R


0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 0.1 0.2 0.3 0.4 0.5 0.6

relative frequency

SF

R

reference SFR moderate strongreference SFR moderate strong

SFRs from identically specified sharpness settingsRadius=0.5 ; Amount=350

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 0.2 0.4 0.6 0.8 1

Spatial frequency

SF

R

Software A Software B


0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 0.2 0.4 0.6 0.8 1

Spatial frequency

SF

R



0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 0.2 0.4 0.6 0.8 1

Spatial frequency

SF

R



0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 0.2 0.4 0.6 0.8 1

Spatial frequency

SF

R



0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 0.2 0.4 0.6 0.8 1

Spatial frequency

SF

R



42

How to Measure Sharpness

• Via the SFR shape and magnitude

• Summary sharpness metrics such as

– Location/magnitude of maximum SFR value

• Visual weighting > acutance

• Sharpness = f(acutance)

0

)( )( dCSFSFRA

0

)( dCSFAr

CMT Acutance

rAAAcutance =

Visual contrast Modulation Transfer (CMT)

Cycle/degree

0

0.5

1

1.5

2

2.5

3

3.5

0 5 10 15 20 25 30 35 40 45


22

43

SFR measurement

Challenges for today’s cameras• Degree of Over-sharpening• Level of SFR field non-uniformity• Aliasing• Texture

Proposed New ISO 12233 Resolution Targets – Rev. 2

CIPA – maximum resolutionNot recommended for SFR

OECF, Slanted EdgesEdge-based SFR Polar sine waves

Sine-based SFR


44

SFR Summary

ISO SFR Method• Based on edge-gradient method• Intended for digital capture devices• Alignment immunity• Localized features• Open source software• Extendable to any angular direction• Ease of target generation; several imaging mechanisms or steps can be

described in terms of spread functionsMTF and SFR

Spread functions and MTF indicate the imaging or transfer of signal detail and sharpness

A number of derivative metrics can be calculated from SFR for process control & monitoring as well as image quality.

For quality-assurance control charts, a good measure of image resolution is the SFR-based sampling efficiency.


23

45

Dead-Leaves Texture MTF Measurement

Method is aimed at providing an effective MTF for image fluctuations (signals) that are influenced by adaptive or signal-dependent image processing

For example adaptive noise cleaning, which could leave edge untouched, but reduce detail in important ‘textured regions’

filterednoisy


46

Dead-leaves target and amplitude spectrum

Noise-free target

Similar to image features Amplitude spectrum


24

47

Dead-Leaves MTF Measurement

1. Transform the captured image array of the target field to one encoded as proportional to luminance.

2. Compute the power-spectral density as the square of the amplitude of the two-dimensional DFT of the array.

3. Divide this array, frequency-by-frequency, by the modeled spectrum for the specific target to yield a two-dimensional array as the square of the effective MTF.

4. Compute the square-root, frequency-by-frequency, of this vector.

5. Compute the one-dimensional MTF vector by a radial-average of this array


48

Dead-leaves target and amplitude spectrum

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-1.5

-1

-0.5

0

0.5

1

cy/pixel

Log

spec

trum

radial average spectrum

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-1.5

-1

-0.5

0

0.5

1

cy/pixel

Log

spec

trum

x

ydiagonal

Cross-sections

• Single result indicates NPS estimation variation

• Cross-section results can indicate direction differences


25

49

NOISE


50

Taxonomy - Noise


26

51

Image Noise

• Noise is a general term applied to any response that detracts from a desired signal. Usually, errors or unwanted fluctuations in images. Given that it can have many sources, it can take several forms.

• Random noise due to film grain or low exposure to a detector. Often looks like “salt and pepper” noise.

• Often reported as an RMS number (standard deviation)

• Structured banding

• Compression artifacts


52

Two-Dimensional Random: Example

025.0R

Fine-grain noise ->high frequency fluctuations

R = print reflectance [0-1]

025.0R

Course-grain noise ->low frequency fluctuations

3 – noise


27

53

Noise

All of these patches have the same average and standard deviation


54

ISO/TC42 Standards for Digital Imaging Noise Metrology

ISO 15739 – Noise for Digital Still CamerasISO 21550 – Dynamic Range for Digital Film Scanners

Commonalities

• Assume an additive noise model; σ2total = σ2

fixed pattern + σ2temporal

• Attempt to discriminate noise contributions with replicate images

• Requires OECF measurement for scene referred metrology

• Dynamic Range reported as a linear ratio, not decibels (dB)

Differences

15739 21550

• Focuses on noise metrology • Focuses on dynamic range metrology

• Uses peak-to-peak definition • Uses Incremental signal definition for

of signal for dynamic range dynamic range calculation

• Provides for frequency weighted • Assumes 100% maximum transmission

visual noise output (informative)

• Assumes 140% luminancemax

• Photoshop plug-in available athttp://www.i3a.org

3 – noise


28

55

Dynamic Range – Cameras and Scanners*

The extent of energy over which a digital capture device can reliably detect signals: reported as either a normalized ratio (xxx:1) or in equivalent optical density units, , R= reflectance.

• ISO 21550 – Dynamic Range for Digital Film Scanners• ISO 15739 – Noise Measurement for Electronic Still Pictures Cameras

Signal Detection → OECF

The mean (average) change in count level associated with a particular input density level.

Reliability of Detection → Noise

The root mean square (std. dev.) count level associated with a particular input density level

High noise ≡ low reliability

Noise

Signal

3 – noise – dynamic range

RD 10log


56

Incremental Signal-to-Noise Ratio – Scanners

Incremental Signal

0

20

40

60

80

100

120

140

160

180

0 0.5 1 1.5 2 2.5 3 3.5 4

Density

Incr

emen

tal S

igna

l Incremental SNR

0.1

1

10

100

1000

0 0.5 1 1.5 2 2.5 3 3.5 4

Density

Incr

emen

tal S

NR

Measured RMS noise (CV)for film scanner

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

0 0.5 1 1.5 2 2.5 3 3.5 4

Density

RM

S no

ise

(CV

)

3 – noise – dynamic range


29

57

Measurement Requirements

Measurements usually require some level of both accuracy and precision.

– Accuracy: average error from a standard

– Precision: variability about the average readingIs your watch set to the correct time?

Factors that influence measurements

– Test target feature location estimation

– Signal processing

– Signal quantization

– Spatial sampling

– Image noise

XXXX

X bias

X

X

X X

X

variation

Burns/Williams, EI2014: Performance Variation

58

Variation

Actual performancee.g., Optical quality across the image field, or with lens zoom position

Measurement errorEstimation of sample statisticse.g. due to signal quantization, sampling

Sampling:variation in estimating parameters (statistics) from test images

Source of imaging performance variability for a digital camera?• Scene exposure.• Image compression parameters.• Camera shake.


30

59

Measurement error example

• Simplest way to quantify measurement variation is by repeating the analysis• This edge SFR example is for one test file and one edge, varying the

location of the region of interest, N = 10

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frequency, cy/pixel

SF

R

Single file, edge (N=10)

ave

+1 std

-1 stdmax

min


60

ISO Protocols vs. Specifications:

ISO Imaging Performance provide protocols on how and what to measure with respect to imaging performance ( e.g. tone, noise, SNR, Resolution, etc.) but …..

The acceptable levels and tolerances for each of those is left to the user because they are largely use-case dependent. Resolution and color aims and tolerances could be quite different whether one is doing cell phone vs. portrait photography

Useful features in software for imaging performance evaluation• Image metadata (header and tag) reading.• Use with multi-attribute test targets.• Testing of results against specified tolerance values.

How & What vs. Pass or Fail


31

61

Test plans – just a few words

• Don’t collect too much data

– Limit the experiment

• Control the variables

– Operator, lighting, zoom position, image processing

• Distinguish benchmarking from monitoring

• Distinguish capability from performance

• Be reasonable with test targets designs

– Any target can be designed to make a camera look bad

• Test under conditions that match expected usage.

• Prioritize the metrics exposure, SFR, noise/artifacts

• For quality-assurance control charts, a good measure of image resolution is the SFR-based sampling efficiency (See Appendix VI)


62

ISO TC42/WG18 Standards Rodeo

PWI - Preliminary Work Item

NP– New Proposal

AWI - Approved new Work Item

WD - Working Draft

CD - Committee Draft

FCD - Final Committee Draft

DIS - Draft International Standard

FDIS - Final Draft International Standard

PRF - Proof of a new International Standard

IS - International Standard

Periodic Review and updates

Stages of an ISO standard

32



PWI - Preliminary Work Item

NP– New Proposal

AWI - Approved new Work Item

WD - Working Draft

CD - Committee Draft

FCD - Final Committee Draft

DIS - Draft International Standard

FDIS - Final Draft International Standard

PRF - Proof of a new International Standard

IS - International Standard

Periodic Review and updates

Stages of an ISO standard



TC42 – PhotographyWG18 – Digital Cameras

12232 Determination of exposure index, ISO speed ratings, standard output sensitivity, and recommended exposure index

14524 Methods for measuring opto‐electronic conversion functions (OECFs)

12233 Electronic still‐picture cameras ‐‐ Resolution measurements

16067-1 Reflective Scanners – Resolution measurements

16067-2 Film Scanners – Resolution measurements

21550 Film scanners ‐ Dynamic range measurements

15739 Noise

The usual suspects for imaging performance

33


12231 - Vocabulary

15781 ‐Measuring shooting time lag, shutter release time lag, shooting rate, and start‐up time

12234 ( multiple parts) ‐ Removable memory

20462 ( multiple parts) ‐ Psychophysical experimental methods for estimating image quality

22028 ‐ Extended colour encodings for digital image storage, manipulation and interchange

IEC 61966 ‐ Colour measurement and management

Important, but not for imaging performance


18841 - Flare measurement techniques for digital camera systems (AWI)

19093 ‐Measuring low light performance (NP)

19567 ‐ Low contrast fine detail measurements

TS 19247 ‐ Guidelines for reliable camera testing

17321 (Parts 1,2 3,& 4) ‐ Colour characterisation of digital still cameras

17850 ‐ Geometric distortion measurements

17957 ‐ Shading measurements

18383 ‐ Specification guideline

19084 ‐ Lateral chromatic displacement measurement

New Kids on the Block

Under development

34

Burns/Williams, EI2014: Standards Rodeo 67

What’s missing ?

Depth of Field / Focus ?

White Balance ?

Sharpening ?

Aliasing Potential ?

Suggestions?

68


TC42 – PhotographyWG18 – Digital Cameras

12232 Determination of exposure index, ISO speed ratings, standard output sensitivity, and recommended exposure index

14524 Methods for measuring opto‐electronic conversion functions (OECFs)

12233 Electronic still‐picture cameras ‐‐ Resolution measurements

16067-1 Reflective Scanners – Resolution measurements

16067-2 Film Scanners – Resolution measurements

21550 Film scanners ‐ Dynamic range measurements

15739 Noise

The usual suspects for imaging performance

Burns/Williams, EI2014: Standards Rodeo

35

69

12231 - Vocabulary

15781 ‐Measuring shooting time lag, shutter release time lag, shooting rate, and start‐up time

12234 ( multiple parts) ‐ Removable memory

20462 ( multiple parts) ‐ Psychophysical experimental methods for estimating image quality

22028 ‐ Extended colour encodings for digital image storage, manipulation and interchange

IEC 61966 ‐ Colour measurement and management

Important, but not for imaging performance


70

18841 - Flare measurement techniques for digital camera systems (AWI)

19093 ‐Measuring low light performance (NP)

19567 ‐ Low contrast fine detail measurements

TS 19247 ‐ Guidelines for reliable camera testing

17321 (Parts 1,2 3,& 4) ‐ Colour characterisation of digital still cameras

17850 ‐ Geometric distortion measurements

17957 ‐ Shading measurements

18383 ‐ Specification guideline

19084 ‐ Lateral chromatic displacement measurement

New Kids on the Block

Under development


36

71

What’s missing ?

Depth of Field / Focus ?

White Balance ?

Sharpening ?

Aliasing Potential ?

Suggestions?



Where do the Standards align


37


Wrap Up

1. Technology used influences imaging performance

2. Imaging performance measurement is standardized for

– Large area image capture (OECF)

– Image detail capture (SFR and MTF)

– Image noise and artifacts (rms, dynamic range)

3. Visual inspection is often needed for artifacts

4. Understanding of technology modes helps interpretation of results. Area arrays, image formatting, software settings, etc.

5. Useful features in software for imaging performance evaluation

• Image metadata (header and tag) reading.

• Use with multi-attribute test targets.

• Testing of results against specified tolerance values.

Send questions to Don [email protected]

Or Peter [email protected]


Appendix I: Important Imaging Characteristics- an imaging performance taxonomy for classifying the metrics -


38


Color – Its just an encoding

Same colors – different encoding

Appendix II: Digital Cameras do not Reproduce Colors, they Encode them.

Reproduction requires interpreting (decoding) the pixel values to display the image for particular audience or purpose.


Appendix III: Spatial Frequency Units

pixel

cyN

ph

pixel

pixel

cy

ph

lp Npixelsph

line-pairs/picture height – Video, digital systems4 -

3 – cycles/pixel – digital processing, filter design

5 – line widths per picture height, LW/PH – Video,12233 standard

6 – cycles/degree (CPD) – visual acuity, image quality

mm

cy

mm

lp

mm

cy

mm

lp

mm

l

22lines/mm = or

cycles/mm, cy/mm, mm-1 - Engineering, film photography

line-pairs/mm - Graphics

1 -

2 - 1 line width

1 line pair

39


Frequency Unit Conversion Table

LW/PH LP/mm L/mm Cycles/mm Cycles/pixel

LW/PH1

⁄ [ 2 picture height (mm)]

⁄ picture height (mm)

⁄ [ 2 picture height (mm)]

⁄ [ 2 # vert. pixels ]

LP/mm [2 picture

height (mm)]

1 2.0 1 pixel pitch (mm)

L/mm picture height

(mm) 0.5 1 0.5

[pixel pitch (mm) ⁄2]

Cycles/mm [2 picture

height (mm)]

1 2.0 1 pixel pitch (mm)

Cycles/pixel [2 # vert.

pixels] ⁄ pixel pitch

(mm)

[2 ⁄ pixel pitch (mm)]

⁄ pixel pitch (mm) 1

LW/PH = Line width per picture heightLP/mm = Line pairs per millimetreL/mm = Lines per millimetreTo convert from left column units to top row units, use operation at their row/column intersection.(e.g., 5 LP/mm 2.0 = 10.0 L/mm )


Appendix IV: ISO 12233, SFR and MTF

The ISO standard for measuring digital still camera resolutions specified a particular method, based on a modified form of edge gradient analysis (EGA) The ISO standard, however, refers to the camera resolution metric as

Spatial Frequency Response (SFR), rather than an MTF There are three basic reasons for this

– Established photographic standards measure MTFs using other methods (sine waves), and the results often differ from EGA results

– The term MTF has been associated with the measurement of systems or subsystems* for which it (approximately) uniquely describes the signal transfer from input to output). For many electronic imaging systems, however, the results will vary with system and image conditions.

– EGA measurement of a system MTF requires compensation for the input target edge characteristics. While this is possible, the original ISO standard did not require it.

___________________

* Linear systems, or those with combinations of linear and point-wise nonlinear subsystems

40


Appendix V: Limiting Resolution Estimation from SFR

EvaluateSFR

ii SFRf ,Find criticalfrequency

pp SFRf where,

InterpolateSFR

'whereSFR

test file,critical value, f

EvaluateSFR

ii SFRf ,Find criticalfrequency

pp SFRf where,

InterpolateSFR

'whereSFR

test file,critical value, f


Two dimensional example

Example of two-dimensional symmetric SFR and corresponding 50% and 10% response contours

41


Appendix VI: ISO Camera Resolution Efficiency Rating Proposal

Electronic still picture cameras are normally marketed using a “megapixel” value, which normally provides thenumber of effective pixels of the image sensor. This can be confusing or misleading, since it is a value relatedto the number of addressable photoelements on the image sensor, and is not a based on any type of resolutionmeasurement.

There is interest in developing a single-value resolution numeric for reporting the measured resolution of electronicstill picture cameras. One proposed metric is the “resolution efficiency rating.”

1. Determine the maximum resolution in LW/PH for the horizontal (RH), vertical (RV), and +/- 45° (R+45 , R-45 )directions.2. Calculate individual directional efficiencies ( EH , EV , E+45 , E+45 ) by normalizing the maximum resolutions of

item #1 by the captured image’s picture height. If any normalized value is greater than 1.0, assign that value to 1.0.3. Combine E+45 and E-45 efficiencies into an equally weighted average diagonal value ED. 4. Calculate the resolution efficiency rating (ER) as the product of 100, ED and the average of EH and EV.

Resolution Efficiency Rating ER = 100 ( ED (EH + EV) /2 )

Example using the CIPA tool: 2048 pixel high x 3072 pixel wide, 6.0 MPixel camera file

Maximum horizontal resolution : RH = 1970 LW/PH Horizontal Resolution Efficiency: EH= 1970/2048 = 0.96Maximum vertical resolution : RV = 1980 LW/PH Vertical Resolution Efficiency: EV = 1980/2048 = 0.97Maximum +45° resolution : R+45 = 1500 LW/PH Diagonal Resolution Efficiency: ED = (1500 + 1500 )/2 = 0.73Maximum -45° resolution : R-45 = 1500 LW/PH

Resolution Efficiency Rating = 100 [(0.73 (0.96 + 0.97)) /2 ] = 70.6


Sampling efficiency Summary

• Sampling efficiency measure is considered as an extension of thecurrent ISO 12233 standard revision effort.

• Based on the ratio of a 10% SFR-spatial frequency bandwidth to thebandwidth implied by the image sampling alone

• Not intended to include the influence of sampling artifacts and imagenoise

• Provides guidance when considering the level of image signal detaillikely to be delivered by a camera

• Sampling efficiency provides a convenient factor to adjust advertisedvalues to yield an effective megapixel value.

digital camera and scanner performance evaluation: standards...

Documents