lec13, video iii (video coding standards),...

63
Multimedia Systems Multimedia Systems Video III Video III (Video Coding Standards) (Video Coding Standards) Course Presentation Course Presentation (Video Coding Standards) (Video Coding Standards) Mahdi Amiri April 2014 Sharif University of Technology

Upload: others

Post on 14-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Multimedia SystemsMultimedia Systems

Video IIIVideo III

(Video Coding Standards)(Video Coding Standards)

Course PresentationCourse Presentation

(Video Coding Standards)(Video Coding Standards)

Mahdi Amiri

April 2014

Sharif University of Technology

Page 2: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsStandardization OrganizationsStandardization Organizations

Two organizations have dominated video compression

standardization.

ITU-T Video Coding Experts Group (VCEG)

International Telecommunications Union –Telecommunications

Standardization Sector (ITU-T, a United Nations Organization,

Multimedia Systems, Mahdi Amiri, Video IIIPage 1

Standardization Sector (ITU-T, a United Nations Organization,

formerly CCITT), Study Group 16, Question 6.

ISO/IEC Moving Picture Experts Group (MPEG)

International Standardization Organization and International

Electrotechnical Commission, Joint Technical Committee Number

1, Subcommittee 29, Working Group 11.

Page 3: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsDynamicsDynamics

VCEG is older and more focused on conventional (esp. low-delay)

video coding goals (e.g. good compression and packet-loss/error

resilience)

MPEG is larger and takes on more ambitious goals (e.g. “object

oriented video”, “synthetic-natural hybrid coding”, and digital

Multimedia Systems, Mahdi Amiri, Video IIIPage 2

oriented video”, “synthetic-natural hybrid coding”, and digital

cinema)

Sometimes the major organizations team up (e.g. ISO, IEC and ITU

teamed up for both MPEG-2 and JPEG)

Relatively little industry consortium activity (DV and organizations

that tweak the video coding standards in minor ways, such as DVD,

3GPP, 3GPP2, SMPTE, IETF, etc.)

Growing activity for internet streaming media outside of formal

standardization (e.g., Microsoft, Real Networks, Quicktime)

Page 4: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsThe Scope of Picture and Video Coding StandardizationThe Scope of Picture and Video Coding Standardization

A Video standard specifically do not define an encoder; rather, they

define the output that an encoder should produce.

A decoding method is defined in each standard (only the Bitstream

Syntax and Decoding Process are standardized):

e.g. use IDCT, but not how to implement the IDCT.

Multimedia Systems, Mahdi Amiri, Video IIIPage 3

e.g. use IDCT, but not how to implement the IDCT.

Permits optimization beyond the obvious.

Permits complexity reduction for implementability.

Provides no guarantees of Quality - only interoperability.

Ensuring Ensuring interoperabilityinteroperability::

Enabling communication between Enabling communication between

devices made by different devices made by different

manufacturersmanufacturers

Page 5: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationObjectiveObjective

A computer algorithm judges the distortion between

videos

Attempts to model a human observer

There is currently no standard method

Multimedia Systems, Mahdi Amiri, Video IIIPage 4

There is currently no standard method

Page 6: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationObjective Metrics: PSNRObjective Metrics: PSNR

Peak Signal-To-Noise Ratio (PSNR)

Used widely in evaluating coding performance

Purely mathematical difference

Can be tricked quite easily

Multimedia Systems, Mahdi Amiri, Video IIIPage 5

Can be tricked quite easily

Root Mean Squared Root Mean Squared

Error (RMSE)Error (RMSE)

255 = 2^n 255 = 2^n –– 11

n: the number of bits per image samplen: the number of bits per image sample

Page 7: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationPSNR, ExamplePSNR, Example

Multimedia Systems, Mahdi Amiri, Video IIIPage 6

OriginalOriginal PSNR 35.4 [dB]PSNR 35.4 [dB] PSNR 29.0 [dB]PSNR 29.0 [dB]

Page 8: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

ABCABAB

OriginalOriginal PSNR 45.53 [dB]PSNR 45.53 [dB]

Multimedia Systems, Mahdi Amiri, Video IIIPage 7

PSNR 36.81 [dB]PSNR 36.81 [dB] PSNR 31.45 [dB]PSNR 31.45 [dB]

Page 9: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationMahalanobisMahalanobis DistanceDistance

PrasantaPrasanta Chandra Chandra MahalanobisMahalanobis

18931893--19721972

Euclid, Euclid, FloruitFloruit 300 BC300 BC

(Statue of Euclid in the Oxford University (Statue of Euclid in the Oxford University

Museum of Natural History)Museum of Natural History)

The The MahalanobisMahalanobis distance differs from Euclidean distance differs from Euclidean

distance in that it takes into account the distance in that it takes into account the

correlations of the data set and is scalecorrelations of the data set and is scale--invariant.invariant.

Multimedia Systems, Mahdi Amiri, Video IIIPage 8

( ) ( )1T −− −x µ Σ x µRef.: www.aiaccess.netRef.: www.aiaccess.net( ) ( )

T

− −x µ x µ

Euclidian distance (Squared)Euclidian distance (Squared) MahalanobisMahalanobis distance (Squared)distance (Squared)

Page 10: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationObjective Metrics: PSNRObjective Metrics: PSNR

How to trick PSNR

Take a natural image

Give more bits to areas you look at more

Give less bits to areas you look at less

Multimedia Systems, Mahdi Amiri, Video IIIPage 9

Give less bits to areas you look at less

Subjective rating will be high, PSNR low

OriginalOriginal Attention Map ExampleAttention Map Example TestTest(High subjective rating, low PSNR)(High subjective rating, low PSNR)

Page 11: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationSubjective: MOSSubjective: MOS

Mean Opinion Score (MOS)

A numerical indication of the perceived quality of the media received

after being transmitted and eventually compressed using codecs. MOS is

expressed in one number, from 1 to 5, 1 being the worst and 5 the best.

A human “subject” rates the video on a scale.A human “subject” rates the video on a scale.

Multimedia Systems, Mahdi Amiri, Video IIIPage 10

Page 12: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationSubjectiveSubjective

In the ITU recommendations, there are many subjective

quality test methods.

Absolute Category Rating (ACR)

Degradation Category Rating (DCR)

Multimedia Systems, Mahdi Amiri, Video IIIPage 11

Degradation Category Rating (DCR)

The Double-Stimulus Continuous Quality-Scale

method (DSCQS)

Page 13: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationSubjective: ACR and DCRSubjective: ACR and DCR

Absolute Category Rating (ACR)

Subjects are asked to rate the quality of the

presentation based on the level of the quality they have

in their opinion for it after viewing or listening it

(Single Stimulus).

No reference sequence.

Multimedia Systems, Mahdi Amiri, Video IIIPage 12

(Single Stimulus).

Degradation Category Rating (DCR)

Test sequences are presented in pairs. The first

stimulus presented in each pair is always the source

reference without any impairments (Double Stimulus).

Ref.: www.irisa.fr/armor/lesmembres/MohamedRef.: www.irisa.fr/armor/lesmembres/Mohamed

/Thesis/node147.html/Thesis/node147.html

Known reference sequence.

Page 14: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Quality EvaluationSubjective: DSCQSSubjective: DSCQS

Double Stimulus Continuous Quality Scale Method (DSCQS)

Multimedia Systems, Mahdi Amiri, Video IIIPage 13

Ref.: www.irisa.fr/armor/lesmembres/MohamedRef.: www.irisa.fr/armor/lesmembres/Mohamed

/Thesis/node147.html/Thesis/node147.html

For having fidelity test the observers are

not told which is the reference sequence.

Unknown reference sequence.

Page 15: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

R-D Curve of Video CodecsABCABC

Multimedia Systems, Mahdi Amiri, Video IIIPage 14

Page 16: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

R-D Curve of Video CodecsABCABC

ReminderReminder

CIFCIF--size image size image

352352××288288

Multimedia Systems, Mahdi Amiri, Video IIIPage 15

Page 17: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

R-D Curve of Video CodecsABCABC

Multimedia Systems, Mahdi Amiri, Video IIIPage 16

Page 18: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

R-D Curve of Video CodecsABCABCR-D Performance of MPEG Codecs

44

46

48

50

Multimedia Systems, Mahdi Amiri, Video IIIPage 17

32

34

36

38

40

42

350 450 550 650 750 850 950 1050

Bit rate (kbps)

PSNR (Y)

MPEG-1 MPEG-2 MPEG-4 H.264

Page 19: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

R-D Curve of Video CodecsABCABC

Multimedia Systems, Mahdi Amiri, Video IIIPage 18

Page 20: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsGary J. Sullivan, Ph.D.

Gary J. Sullivan, Ph.D.

Rapporteur

Rapporteur| C

hair

| Chair

Rapporteur

Rapporteur| Co

| Co--Chair

Chair

Rapporteur

Rapporteur| C

o| C

o--Chair

Chair

Multimedia Systems, Mahdi Amiri, Video IIIPage 19

Gary J. Sullivan, Ph.D.

Gary J. Sullivan, Ph.D.

ITU

ITU--T VCEG

T VCEG Rapporteur

Rapporteur

ISO/IEC M

PEG Video

ISO/IEC M

PEG Video Rapporteur

Rapporteur

ITU/ISO/IEC JVT

ITU/ISO/IEC JVT Rapporteur

Rapporteur

Page 21: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsHistoryHistory

Multimedia Systems, Mahdi Amiri, Video IIIPage 20

en.wikipedia.org/wiki/Data_compressionThe Society of Motion Picture and Television Engineers, SMPTE (pron. simpti) is an internationally

recognized standards organizations founded in 1916 (en.wikipedia.org/wiki/SMPTE).

Page 22: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMajor Video Compression ApplicationsMajor Video Compression Applications

Multimedia Systems, Mahdi Amiri, Video IIIPage 21

Page 23: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMoving Picture Experts Group (MPEG)Moving Picture Experts Group (MPEG)

A working group of ISO/IEC in charge of the

development of standards for coded

representation of digital audio and video and

related data.

Multimedia Systems, Mahdi Amiri, Video IIIPage 22

related data.

Established in 1988

23 years of activity

The number of independent standards: more than

125

Page 24: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEG-1 The standard on which such products as Video CD and MP3 are based

MPEG-2 The standard on which such products as Digital Television set top boxes and DVD are

based;

MPEG-4 The standard for multimedia for the fixed and mobile web;

MPEG-7 The standard for description and search of audio and visual content;

MPEG-21 The Multimedia Framework;

MPEG-A The standard providing application-specific formats by integrating multiple MPEG

technologies;

MPEG-B A collection of Systems specific standards

MPEG-C A collection of Video specific standards

Multimedia Systems, Mahdi Amiri, Video IIIPage 23

MPEG-C A collection of Video specific standards

MPEG-D A collection of Audio specific standards

MPEG-E A standard (M3W) providing support to download and execution of multimedia

applications

MPEG-H A standard (HEVC) providing a significantly increased video compression performance

MPEG-M A standard (MXM) for packaging and reusability of MPEG technologies

MPEG-U A standard for rich-media user interface

MPEG-V A standard for interchange with virtual worlds

Page 25: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsVideo Coding Experts Group (VCEG)Video Coding Experts Group (VCEG)

Part of study group 16 (Multimedia coding, systems and

applications) of the ITU-T. Established in 1984

H.120 The first digital video coding standard

H.261 Was the first practical digital video coding standard.

Multimedia Systems, Mahdi Amiri, Video IIIPage 24

H.261 Was the first practical digital video coding standard.

H.262 It is identical in content to the video part of the ISO/IEC MPEG-2 standard.

H.263 Provided a suitable replacement for H.261 at all bitrates.

H.263v2 Also known as H.263+, Enhanced robustness against data loss in the transmission channel.

H.264 The ITU-T H.264 standard and the ISO/IEC MPEG-4 Part 10 standard (formally, ISO/IEC 14496-

10) are technically identical.

H.265 Not yet developed; expected 2012 or later.

H.271 Video back channel messages for conveyance of status information and requests from a video

receiver to a video sender.

Page 26: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsH.120H.120

The First Digital Video Coding Standard

ITU-T (ex-CCITT) Rec. H.120: 1984

v1 (1984) had conditional replenishment, DPCM, scalar

quantization, variable-length coding, switch for quincunx sampling

v2 (1988) added motion compensation and background prediction

Multimedia Systems, Mahdi Amiri, Video IIIPage 25

v2 (1988) added motion compensation and background prediction

Operated at 1.544 (NTSC) and 2.048 (PAL) Mbits/s

Few units made, essentially not in use today

Conditional ReplenishmentConditional Replenishment: Can signal to leave a block area of the image : Can signal to leave a block area of the image

unchanged, or replace it with new data (using a threshold value).unchanged, or replace it with new data (using a threshold value).

Quincunx samplingQuincunx sampling: In a digital video system, a sampling structure with : In a digital video system, a sampling structure with

an array of samples where alternate rows of pixel samples are displaced an array of samples where alternate rows of pixel samples are displaced

horizontally in the grid by half of the pitch of the pixel samples along the horizontally in the grid by half of the pitch of the pixel samples along the

remaining rows.remaining rows.

Page 27: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsH.261H.261

ITU-T, completed in 1990, The first widespread

practical success

Video telephony and teleconferencing over ISDN

(Integrated Services Digital Network)

Multimedia Systems, Mahdi Amiri, Video IIIPage 26

Embodying typical structure that dominates today

Combination of DPCM and DCT

Motion Compensation

p x 64kbps (64-2048 kbps)

Still in use, although mostly as a backward-compatibility

feature – overtaken by H.263

Page 28: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--11

MPEG-1: “Coding of moving pictures and associated audio for

digital storage media” (1992)

Group of Pictures, Motion Estimation and Motion Compensation,

Differential Coding, DCT, Quantization, Entropy Coding

Video on digital storage media (CD-ROM)

Multimedia Systems, Mahdi Amiri, Video IIIPage 27

Video on digital storage media (CD-ROM)

Target was VHS Quality at 1.5MBits/s (at 352x240 resolution)

Basis of Video-CD

MP3 (MPEG-1 Layer 3)

16 bits

Sampling rate - 32, 44.1, or 48 kHz

Bitrate – 32 to 320 kbps

De facto - 44.1 kHz sample rate, 192 kbps bitrate

Page 29: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--11

Only supports progressive pictures.

Adds bi-directional motion prediction to H.261 design.

Adds half-pixel motion estimation (See next slide).

Slice-structured coding.

Multimedia Systems, Mahdi Amiri, Video IIIPage 28

Slice-structured coding.

DC-only “D” pictures.

Superior quality to H.261 when operated a higher bit

rates ( > 1 Mbps for CIF 352x288 resolution).

Now mostly overtaken by MPEG-2.

Page 30: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--1, Half1, Half--Pixel MEPixel ME

Half-Pixel (coarse-fine) Motion Estimation Algorithm

1) Coarse step: Perform integer motion estimation on blocks; find best integer-pixel MV

2) Fine step: Refine estimate to find best half-pixel MV

a) Spatially interpolate the selected region in reference frame

b) Compare current block to interpolated reference frame block

c) Choose the integer or half-pixel offset that provides best matchTypically, bilinear Typically, bilinear

Multimedia Systems, Mahdi Amiri, Video IIIPage 29

Typically, bilinear Typically, bilinear

interpolation is used for interpolation is used for

spatial interpolationspatial interpolation

Page 31: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--22

MPEG-2: “Generic coding of Moving Pictures and

Associated Audio”.

Broadcasting and storage.

Satellite TV, DVD, Digital TV.

Multimedia Systems, Mahdi Amiri, Video IIIPage 30

Ubiquity in hardware implies that it will be here for a

long time.

Transition to HDTV has taken over 10 years and is not finished yet

Different profiles and levels allow for quality control.

Bitrates: Typ. 4-9 MBits/s (Not especially useful below 4 Mbps,

normal range of use 5-30 Mbps).

Page 32: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--22

Support for interlaced scan, various picture

sampling formats, user defined quantization

matrix.

Essentially same as MPEG-1 for progressive-

Multimedia Systems, Mahdi Amiri, Video IIIPage 31

Essentially same as MPEG-1 for progressive-

scan pictures.

Various forms of scalability (SNR, Spatial,

Temporal and hybrid).

Base Layer: Basic quality requirement, For SDTV.

Enhanced Layer: High quality service, For HDTV.

Page 33: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--2 Profiles and Levels2 Profiles and Levels

Goal: To enable more efficient implementations for different

applications (interoperability points)

Profile: Subset of the tools applicable for a family of applications

Level: Bounds on the complexity for any profile

Multimedia Systems, Mahdi Amiri, Video IIIPage 32

Page 34: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsBitrateBitrate allocationallocation

CBR – Constant BitRate

Streaming media uses this.

Easier to implement.

VBR – Variable BitRate

Multimedia Systems, Mahdi Amiri, Video IIIPage 33

VBR – Variable BitRate

DVD’s use this.

Allocate more bits for complex scenes.

Usually requires 2-pass coding.

This is worth it, because you assume that you encode

once, decode many times.

Page 35: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEG Container FormatMPEG Container Format

Container format is a file format that can

contain data compressed by standard codecs.

2 types for MPEG

Program Stream (PS) – Designed for

Multimedia Systems, Mahdi Amiri, Video IIIPage 34

Program Stream (PS) – Designed for

reasonably reliable media, such as disks.

Transport Stream (TS) – Designed for lossy

links, such as networks or broadcast antennas.

Page 36: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--3 ?3 ?

Originally developed for HDTV, but abandoned when

MPEG-2 was determined to be sufficient.

Multimedia Systems, Mahdi Amiri, Video IIIPage 35

Page 37: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsH.263H.263

ITU-T Rec. H.263 (v1: 1995): The next generation of

video coding performance, developed by ITU-T – the

current premier ITU-T video standard (has overtaken

H.261 as dominant videoconferencing codec)

Video telephony over PSTN (public switched telephone

Multimedia Systems, Mahdi Amiri, Video IIIPage 36

Video telephony over PSTN (public switched telephone

network)

Wins by a factor of two at very low rates

Version 2 (late 1997 / early 1998) & version 3 (2000)

later developed with a large number of new features

H.263+ & H.263++ (Extensions to H.263)

Page 38: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--44

MPEG-4: “Coding of audio-visual objects”.

Started as very low-bitrate project.

Contains the H.263 baseline design and adds many

creative new extras:

Multimedia Systems, Mahdi Amiri, Video IIIPage 37

creative new extras:

Coding of media objects (Segmented coding of shapes).

Bitrate: variable.

Synthetic/Semi-synthetic objects.

XMT: Like HTML, but to build videos.

First standard with Intellectual Property Management.

Page 39: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--44

Part Number Title Description

Part 1 ISO/IEC 14496-1 SystemsDescribes synchronization and multiplexing of video and audio. For example

Transport stream.

Part 2 ISO/IEC 14496-2 VisualA compression codec for visual data (video, still textures, synthetic images, etc.).

One of the many "profiles" in Part 2 is the Advanced Simple Profile (ASP).

Part 3 ISO/IEC 14496-3 Audio

A set of compression codecs for perceptual coding of audio signals, including some

variations of Advanced Audio Coding (AAC) as well as other audio/speech coding

tools.

Part 4 ISO/IEC 14496-4 Conformance Describes procedures for testing conformance to other parts of the standard.

Part 5 ISO/IEC 14496-5 Reference Software Provides software for demonstrating and clarifying the other parts of the standard.

Delivery Multimedia

Multimedia Systems, Mahdi Amiri, Video IIIPage 38

Part 6 ISO/IEC 14496-6

Delivery Multimedia

Integration Framework

(DMIF).

Part 7 ISO/IEC 14496-7Optimized Reference

Software

Provides examples of how to make improved implementations (e.g., in relation to

Part 5).

Part 8 ISO/IEC 14496-8 Carriage on IP

networksSpecifies a method to carry MPEG-4 content on IP networks.

Part 9 ISO/IEC 14496-9 Reference HardwareProvides hardware designs for demonstrating how to implement the other parts of

the standard.

Part 10 ISO/IEC 14496-

10

Advanced Video

Coding (AVC)A codec for video signals which is technically identical to the ITU-T H.264 standard.

http://en.wikipedia.org/wiki/MPEGhttp://en.wikipedia.org/wiki/MPEG--44

Page 40: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--4, Object Based Coding4, Object Based Coding

Extension of MPEG-1/2-type algorithms to code arbitrarily shaped objects

Multimedia Systems, Mahdi Amiri, Video IIIPage 39

Basic Idea: Extend BlockBasic Idea: Extend Block--DCT and BlockDCT and Block--ME/MCME/MC--

prediction to code prediction to code arbitrarily shaped arbitrarily shaped objects.objects.

[MPEG Committee][MPEG Committee]

Page 41: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--4, Sprite Coding4, Sprite Coding

Sprite: Large background image.

Hypothesis: Same background exists for many

frames, changes resulting from camera motion and

occlusions.

Multimedia Systems, Mahdi Amiri, Video IIIPage 40

One possible coding strategy:

1. Code & transmit entire sprite once.

2. Only transmit camera motion parameters for each

subsequent frame.

Significant coding gain for some scenes.

Page 42: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsMPEGMPEG--4, Sprite Coding4, Sprite Coding

Multimedia Systems, Mahdi Amiri, Video IIIPage 41

[MPEG Committee][MPEG Committee]

Page 43: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsH.264 or MPEGH.264 or MPEG--4 Part 10 (AVC)4 Part 10 (AVC)

MPEG-4 Part 10: Advanced Video Coding / H.264.

Designed by a Joint MPEG and VCEG.

Claims 50% bitrate savings to MPEG-2, 30% over MPEG-4!

Bitrate: 10’s to 100’s kb/s.

Variable Block Size, Multiple Reference Frames, Integer Transform,

Multimedia Systems, Mahdi Amiri, Video IIIPage 42

Variable Block Size, Multiple Reference Frames, Integer Transform,

Intra Prediction, In-loop Deblocking Filtering, 1/4-pel Resolution

Motion Estimation, ASO (Arbitrary Slice Ordering), FMO (Flexible

Macroblock Ordering).

Enhanced entropy coding.CAVLC (Context Adaptive Variable Length Coding)

CABAC (Context Adaptive Binary Arithmetic Codes)

Increased complexity relative to prior standards.

Page 44: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding Standards, H.264Integer TransformInteger Transform

MPEG-2, MPEG-4

Multimedia Systems, Mahdi Amiri, Video IIIPage 43

MPEG-4 AVC

Find out more about integer

transform by looking for

"lifting scheme"

Page 45: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding Standards, H.264Variable Block SizeVariable Block Size

The fixed block size may not be suitable for all motion objects

Improve the flexibility of comparison

Reduce the error of comparison

7 types of blocks for selection

Multimedia Systems, Mahdi Amiri, Video IIIPage 44

00

10

2 3

0 11

16 x 16 16 x 8 8 x 16 8 x 8

00

10

2 3

0 11

8 x 8 8 x 4 4 x 8 4 x 4

Page 46: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding Standards, H.264Variable Block SizeVariable Block Size

Residual (without MC) showing optimum choice of partitions

Multimedia Systems, Mahdi Amiri, Video IIIPage 45

Page 47: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding Standards, H.264Multiple Reference FramesMultiple Reference Frames

The neighboring frames are not the most similar in some cases.

The B-frame can be reference frame.

B-frame is close to the target frame in many situations.

Multimedia Systems, Mahdi Amiri, Video IIIPage 46

Page 48: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding Standards, H.264DeblockingDeblocking FilterFilter

There are severe blocking artifacts.

4*4 transforms and block-based motion compensation.

Result in bit rate savings of around 6~9%.

Improve subjective quality and PSNR of the decoded picture.

Multimedia Systems, Mahdi Amiri, Video IIIPage 47

Without Filter With AVC Deblocking Filter

Page 49: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding Standards, H.264FMO (Flexible FMO (Flexible MacroblockMacroblock Ordering)Ordering)

Slice (composed in FMO) ���� Enhance Robustness to data loss

Multimedia Systems, Mahdi Amiri, Video IIIPage 48

Subdivision of a picture into Subdivision of a picture into

slices when slices when not using FMOnot using FMO

Subdivision of a QCIF frame into Subdivision of a QCIF frame into

slices when slices when utilizing FMOutilizing FMO

Page 50: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

H.264, ProfilesABCABC

ABC

http://en.wikipedia.org/wiki/MPEG

http://en.wikipedia.org/wiki/MPEG--4_AVC.htm

4_AVC.htm

Multimedia Systems, Mahdi Amiri, Video IIIPage 49

http://en.wikipedia.org/wiki/MPEG

http://en.wikipedia.org/wiki/MPEG

Page 51: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

H.264, ProfilesABCABC

ABC

Multimedia Systems, Mahdi Amiri, Video IIIPage 50

Page 52: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding Standards, H.265H.265 or HEVCH.265 or HEVC

High Efficiency Video Coding (HEVC) / H.265.

Under joint development by MPEG and VCEG.

Claims 50% bitrate savings to H.264.

Two of the key features where HEVC was improved compared to H.264/MPEG-4

AVC was:

Support for higher resolution video.

MP: Main Profile

HP: High Profiel

ASP: Advanced Simple Profile

HLP: High Latency Profile

Multimedia Systems, Mahdi Amiri, Video IIIPage 51

Support for higher resolution video.

Improved parallel processing methods. en.wikipedia.org/wiki/High_Efficiency_Video_Coding

Page 53: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Scalable Video CodingMotivationMotivation

Basic situation:

1. Diverse receivers may request the same video

Different bandwidths, spatial resolutions, frame rates, computational capabilities

2. Heterogeneous networks and a priori unknown network conditions

Wired and wireless links, time-varying bandwidths

When you originally code the video you don’t know which client or network situation

Multimedia Systems, Mahdi Amiri, Video IIIPage 52

When you originally code the video you don’t know which client or network situation

will exist in the future.

Probably have multiple different situations, each requiring a different compressed

bitstream.

Need a different compressed video matched to each situation.

Possible solutions:

1. Compress & store MANY different versions of the same video.

2. Real-time transcoding (e.g. decode/re-encode).

3. Scalable coding.

Page 54: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Scalable Video CodingType of ScalabilityType of Scalability

Multimedia Systems, Mahdi Amiri, Video IIIPage 53

The basic types of scalability in video codingThe basic types of scalability in video coding

Page 55: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Scalable Video CodingTemporal ScalabilityTemporal Scalability

Based on the use of B-frames to refine the temporal resolution.

B-frames are dependent on other frames.

However, no other frame depends on a B-frame.

Each B-frame may be discarded without affecting other frames.

Multimedia Systems, Mahdi Amiri, Video IIIPage 54

Page 56: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Scalable Video CodingSpatial ScalabilitySpatial Scalability

Based on refining the spatial resolution

Base layer is low resolution version of video.

Enhanced (Enh1) contains coded difference between

upsampled base layer and original video.

Also called: Pyramid coding.

Multimedia Systems, Mahdi Amiri, Video IIIPage 55

Page 57: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Scalable Video CodingQuality ScalabilityQuality Scalability

Based on refining the amplitude resolution.

Base layer uses a coarse quantizer.

Enh1applies a finer quantizer to the difference between the original DCT

coefficients and the coarsely quantized base layer coefficients.

Also called: SNR Scalability.

Multimedia Systems, Mahdi Amiri, Video IIIPage 56

Note: Base & enhancement layers Note: Base & enhancement layers

are at the are at the samesame spatial resolution.spatial resolution.

Page 58: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Video Coding StandardsIndependent playersIndependent players

Efforts from independent companies in video

compression.

Main competitors:RealNetworks (www.real.com)

Multimedia Systems, Mahdi Amiri, Video IIIPage 57

RealNetworks (www.real.com)

Microsoft Windows Media (WM) (www.microsoft.com)

Apple Quicktime (www.apple.com)

On2 (www.on2.com) (now acquired by Google)

Page 59: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

MPEG & Multimedia ApplicationMPEGMPEG--77

MPEG-7: “Multimedia Content Description

Interface” (2001)

Describing audio/video

Applications:

It uses XML to store metadata, and can

be attached to timecode in order to tag

particular events, or synchronize lyrics

Multimedia Systems, Mahdi Amiri, Video IIIPage 58

Applications:

Indexing of video databases.

Search & Retrieval.

Browsing.

Independence between description and content.

particular events, or synchronize lyrics

to a song, for example.

Page 60: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

MPEGMPEG--2121

Aims at defining an open framework for multimedia

applications.

MPEG-21 is based on two essential concepts:

Definition of a Digital Item (a fundamental unit of distribution

and transaction).

MPEG & Multimedia Application

Multimedia Systems, Mahdi Amiri, Video IIIPage 59

and transaction).

Users interacting with Digital Items.

Main objective of the MPEG-21:

Is to define the technology needed to support users to exchange,

access, consume, trade or manipulate Digital Items in an efficient

and transparent way.

Page 61: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

MPEGMPEG--21 and MPEG21 and MPEG--7 Example7 Example

Metadata and Identifiers

within an MPEG-21 Music

Album

MPEG & Multimedia Application

Multimedia Systems, Mahdi Amiri, Video IIIPage 60

Page 62: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

MPEGMPEG--21, DIDL21, DIDL

Digital Item Declaration Language (DIDL)

Example:

<ITEM>

<DESCRIPTOR>

MPEG & Multimedia Application

Multimedia Systems, Mahdi Amiri, Video IIIPage 61

<DESCRIPTOR>

<STATEMENT TYPE=“text/plain”>

A Personal Photo

</STATEMENT>

</DESCRIPTOR>

<COMPONENT>

<RESOURCE

REF=“file:///pix/dsc1234.REF=“file:///pix/d

sc1234.jpg”/>

</COMPONENT>

</ITEM>

The Digital Item Declaration

Model describes a set of

abstract terms and concepts

to form a useful model for

defining Digital Items.

Page 63: Lec13, Video III (Video Coding Standards), v1.06.pptce.sharif.edu/courses/92-93/2/ce342-1/resources... · Video Quality Evaluation Subjective: ACR and DCR Absolute Category Rating

Thank You

Multimedia SystemsMultimedia Systems

Video III (Video Coding Standards)Video III (Video Coding Standards)

Multimedia Systems, Mahdi Amiri, Video IIIPage 62

Thank You

1. http://ce.sharif.edu/~m_amiri/

2. http://www.dml.ir/

FIND OUT MORE AT...

Next Session: Multimedia Networks INext Session: Multimedia Networks I