IMAGE QUALITY ASSESSMENT

H. R. Sheikh, A. C. Bovik, “Image Information and Visual Quality,” IEEE Trans. Image

Process., vol. 15, no. 2, pp. 430-444, Feb. 2006

Lab for Image and Video Engi., Dept. of ECE

Univ. of Texas at Austin

Outline

Introduction of Image Quality Assessment

Visual Information Fidelity Experiments and Results Conclusion

Quality Assessment (QA)

For testing, optimizing, bench-marking, and monitoring applications.

Quality?

Three Broad QA Categories

Full-Reference (FR) QA Methods Non-Reference (NR) QA Methods Reduced-Reference (RR) QA Methods

Reference Image

Distorted ImageFR QA Quality

PSNR Simple but not close to human visual

quality

Contrast enhancement

Blurred

JPEG compressed

VIF = 1.10

VIF = 0.07VIF = 0.10

Prior Arts

Image Quality Assessment based on Error Sensitivity

CSF: Contrast Sensitivity FunctionChannel Decomposition: DCT or Wavelet TransformError Normalization: Convert the Error into Units of Just Noticeable Difference (JND)Error Pooling:

1

,,

l kklkl eeE

Problems of Error-Sensitivity Approaches

The Quality Definition Problem The Suprathreshold Problem The Natural Image Complexity Problem The Decorrelation Problem The Cognitive Interaction Problem

Visual Information Fidelity

Natural Image Source

Channel(Distortion)

HVS

HVS

C DF

E

ReferenceImage

TestImage

Human Visual

System

Reference Image Information

Human Visual

System

Test Image Information

Definition of VIF

subbandsj

jNjNjN

subbandsj

jNjNjN

sECI

sFCI

VIF,,,

,,,

;

;

Natural Image Source

Channel(Distortion)

HVS

HVS

C DF

E

Source Model

The natural images are modeled in the wavelet domain using Gaussian scale mixtures (GSMs).

tscorfficien bubband toingcorrespond

vectorsldimensiona- are and

: , : and

indices spatial ofset thedenodes where

)C covariance andmean -(zero

:

U2

M

MUC

IiUUIiSS

I

S

IiUSUSC

ii

ii

i

ii

The subband coefficients are partitioned into nonoverlapping blocks of M coefficients each

Gaussian scale mixture (GSM)

Wavelet coefficient => non-Gaussian

The variance is proportional to the squared magnitudes of coefficients at spatial positions.

UzX

V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a local Gaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol. 4119, pp. 363–371, 2000.

Implementation Issues

Assumption about the source model:

N

i

TiiU

iUTi

i

CCN

C

M

CCCs

1

12

1ˆ

ˆ

V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a localGaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol.4119, pp. 363–371, 2000.

Distortion Model

IC

IiVV

IigG

IiVCgVGCD

vV

i

i

iii

2 ance with variRF noise

Gaussian mean -zreo additive stationary a is :

fieldgain scalar icdeterminst a is : where

:

Distorted Images

Distorted Images Synthesized versions

Distorted Images

Human Visual System (HVS)Model

Natural Image Source

Channel(Distortion)

HVS

HVS

C DF

E

ICC

C

IiNNIiNN

NDF

NCE

nNN

i

ii

2'

aslity dimensiona same the

ithGaussian w temultivaria eduncorrelatmean -zreo are

:'' and : where

image)(test '

image) (reference

Visual Information Fidelity Criterion (IFC)

Natural Image Source

Channel(Distortion)

HVS

HVS

C DF

E

C E

Mutual InformationI(C;E)

Mutual Information

Assuming that G, and are known2v 2

n

N

i n

nUi

N

iiiiii

N

iiji

Njiji

N

j

N

i

NNN

I

ICs

sNhsNCh

sECI

sECECIsECI

12

22

2

1

1

11

1 1

log2

1

;

,,;;

C E

Mutual InformationI(C;E)

Mutual Information

N

i

M

k nv

kiiNNN

N

i

M

k n

ki

N

i n

nUiNNN

k

TUU

sgsFCI

s

I

ICssECI

QQCC

1 122

22

2

1 12

2

2

12

22

2

1log2

1;

1log2

1

log2

1;

seigenvalue ofmatrix diagonal a is

symmetric, is Since

Implementation Issues

Assumption about the source model:

N

i

TiiU

iUTi

i

CCN

C

M

CCCs

1

12

1ˆ

ˆ

V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a localGaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol.4119, pp. 363–371, 2000.

Implementation Issues

Assumption about the distortion model:

use B x B window centered at coefficient i to estimate and at i

Assumption about the HVS model:

Hand-optimize the value of

),(ˆ),(ˆ

),(),(ˆ2,

1

DCCovgDDCov

CCCovDCCovg

iiv

i

ig2ˆv

2ˆn

(by linear regression)

Definition of VIF

subbandsj

jNjNjN

subbandsj

jNjNjN

sECI

sFCI

VIF,,,

,,,

;

;

Natural Image Source

Channel(Distortion)

HVS

HVS

C DF

E

Experiments Twenty-nine high-resolution(768x512) 24-bits/pixel

RGB color images Five distortion types: JPEG 2000, JPEG, white

noise in RGB components, Gaussian blur, and transmission errors

20-25 human observers Perception of quality: “Bad,” “Poor,” “Fair,” “Good,”

and “Excellent” Scale to 1-100 range and obtain the difference

mean opinion score (DMOS) for each distorted image

Data base: http://live.ece.utexas.edu/research/quality/

Scatter Plots for Four Objective Quality Criteria

(x) JPEG2000, (+) JPEG,(o) white noise in RGB space, (box) Gaussian blur, and (diamond) transmission Errors in JPEG2000 stream over fast-fading Rayleigh channel

Scatter Plots for the Quality Prediction

Validation Scores

THE VALIDATION CRITERIA ARE: CORRELATION COEFFICIENT (CC), MEAN ABSOLUTE ERROR (MAE), ROOT MEAN-SQUARED ERROR (RMS), OUTLIER RATIO(OR), AND SPEARMAN RANK-ORDER CORRELATION COEFFICIENT (SROCC)

Two version of VIF:VIF using the finest resolution at all orientations andUsing the horizontal and vertical orientations only

Cross-Distortion Performance

Cross-Distortion Performance

(dark solid) JPEG2000, (dashed) JPEG, (dotted) white noise, (dash-dot) Gaussian blur, and (light solid) transmission errors in JPEG2000 stream over fast-fading Rayleigh channel

Dependence on the HVS Parameter

Dependence of VIF performance on the parameter.(solid) VIF, (dashed) PSNR,(dash-dot) Sarnoff JNDMetrix 8.0, and (dotted) MSSIM.

2ˆn

Conclusion

A VIF criterion for full-reference image QA is presented.

The VIF was demonstrated to be better than a state-of-the-art HVS-based method, the Sarnoff’s JND-Metrix, as well as a state-of-the-art structural fidelity criterion, the SSIM index

The VIF provides the ability to predict the enhanced image quality by contrast enhancement operation.

Reference1. H. R. Sheikh, A. C. Bovik, “Image Information and Visual

Quality,” IEEE Trans. Image Process., vol. 15, no. 2, pp. 430-444, Feb. 2006.

2. H. R. Sheikh, A. C. Bovik, and G. de Veciana, “An information fidelity criterion for image quality assessment using natural scene statistics,” IEEE Trans. Image Process., vol. 14, no. 12, pp. 2117–2128, Dec. 2005.

3. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error measurement to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.

4. V. Strela, J. Portilla, and E. Simoncelli, “Image denoising using a local Gaussian scale mixture model in the wavelet domain,” Proc. SPIE, vol. 4119, pp. 363–371, 2000.