1

Video Codec Evaluation

Inf 5080

Andreas Feuersinger

Overview of presentation

MotivationObjective approaches (algorithms)

PSNR, VQM, SSIMSubjective approaches (human eye)

SAMVIQ, DSIS, DSCQS, SSCQE, SDSCE Expamples

Comparison of evaluationsConclusion

2

Motivation

Acquisition, processing, compression, storage, transmission, reproduction and any variety of distortion which results in a degradation of visual qualityLots of codecs which are all supposed to be the “best”Question: How to measure video quality?

Objective approaches

PSNR (Peak Signal to Noise ratio)“easy”, straight forward approachLogarithmic value of the inverted Mean Square Error

VQM (Video Quality Metric) [2]1991 developed with up to 95% correlation to subjective approachesDividing of temporal an spatial informationA convolution an standard deviation gives then grade of distortionDifferent models (television, videoconferencing, general, ...)

3

Objective approaches - SSIMStructural SIMilarity (SSIM) [1]“The measurement of structural information loss can

provide a good approximation to perceived image distortion.”

Subjective approaches “Golden Device” of video quality is used (human eye)For comparison reasons there are standards defined by ITU (International Telecommunication Union)Standards regulate

strict viewing conditionstype of displays usedsource signal selected15 or more non-expert observerstest session should not last more than half an hourlength of each clip is recommended to be 5 or 10 secondsdefined grading scale

4

Subjective approaches

Standardized by ITU-R BT.500 & ITU-T P.910

DSIS: Double Stimulus Impairment ScaleDSCQS: Double Stimulus Continuous Quality ScaleSSCQE: Single Stimulus Continuous EvaluationSDSCE: Simultaneous Double Stimulus for Continuous EvaluationSAMVIQ: Subjective Assessment Methodology for Video Quality

SAMVIQ [5]Developed by EBU-Group B/VIMMembers:

IRT (Institut für Rundfunktechnik)FTRD (France Telekom R&D)RAI (Radio Audizioni Italiane)NRK (Norsk Rikskringkasting)

Start 2001, Finished 20042001-2002: Development2002-2003: Evaluation of Codecs (CIF,QCIF)2004:Testing

5

SAMVIQ - Motivationprevious methods created for TVdifferences

Features

Featurelist of standards

6

Expamples

CS MSU Graphics&Media Lab Video Group (2006)

Nokia test for handhelds

SC MSU (2006) [6]

DivX 6.0XviD 1.1.0x264 svn-352MWMV 9.0

SAMVIQ as reference for objective measurements

7

CS MSU (2006) testscenes

CS MSU (2006) testscenes

8

CS MSU (2006) results

CS MSU (2006) resultsSubjective against objective graphsSequences must be treated separately Sequence is on a line > good correlation subj. and obj. (like SSIM)

9

Nokia single stimulus method [4]Nokia’s SS test ofH.264/AVC and WMV 9(Mustonen et al, 2003)

H.264 better – especiallyin the lower bitrates

Conclusion

“Golden device” very difficult to model Objective models useful for quick, cost effective testsReliable and accurate results can only be made by “real eyeballs”

10

References[1] http://www.cns.nyu.edu/~lcv/ssim/[2] http://www.its.bldrdoc.gov/n3/video/pdf/vqm_techniques_v2.pdf[3] http://www.ebu.ch/trev_301-samviq.pdf[4] http://people.arcada.fi/~johnny/videocodec.pdf[5] http://www.irt.de/IRT/veranstaltungen/irt-tech-wiss-koll-video-

qualitaet.pdf[6] http://www.compression.ru/video/codec_comparison/pdf/-

msu_subjective_codecs_comparison_en.pdf