3. av compression - more about video compression & mpeg - dec06

8/4/2019 3. AV compression - More about video compression & MPEG - Dec06

1/39

Audio/Video compressionMore about video compression and MPEG

Alain Bouffioux

December, 20, 2006


2/39

December, 20, 2006AV Compression / Alain Bouffioux

2

Agenda

About video formatsVideo compression

Audio/Video synchronisation

The MPEG model

The MPEG model and its situation in a communication/storage

context


3/39


3

Agenda



The MPEG model


context


4/39


4

Some video formats (1)

Max. component video signal bandwidth: 6 MHz.

CCIR601 (CCIR is now ITU-R): Video sampling frequency: 13.5 MHz

for 525 & 625 line standards

(Shannon requirement)

Synchronous with line (& image) sampling frequency

Fsampling= 864*Fh for 625 line system (50Hz countries)

Fsampling= 858*Fh for 525 line system (60Hz countries)

Why synchronous? Points at the same place

RGB format


5/39


5


YCbCr format

Cb = B-Y, Cr = R-Y

Eye is more sensitive to luminance than to chrominance (lower

resolution needed for chrominance)

Red

Blue

Green

MatrixLPF

Y

Cb

Cr

ModulatorSubC

+CompositeVideo


6/39


6


The 4:2:2 format Y sampling @ 13.5 MHz

C sampling @ 6.75 MHz

8 bits per pixel

720 active points per line

576 lines active lines per image (2 fields) (625 lines)

and 480 active lines (525 lines) Pixels are not square (e.g. for 480 lines, only 640 active points are needed

- VGA format)

Image size 720*576 or 720*480

The 4:2:0 format

Vertical chrominance resolution reduced by a factor 2(average on two successive lines)


7/39December, 20, 2006AV Compression / Alain Bouffioux

7


SIF format (Source Intermediate Format)

Half the vertical & horizontal resolution of 4:2:0

For 50Hz countries: Luminance: 360*288

Chrominance: 180*120

CIF format (Common Intermediate Format) Intermediate format used in videoconferencing

(communication between US & Europe)

resolution: 360*288

Sampling frequency: 30 Hz

QCIF (Quarter CIF) Half the vertical & horizontal resolution of CIF.



8

Agenda



The MPEG model


context



9

Video compression in MPEG-1&2 (1/6)

Principles removal of intra-picture redundancy :Image is decomposed in 8*8 pixels sub-images.

Each sub-image contains redundant information

DCT transformation (in frequency domain) de-correlates the input

signal.( most energy in low spatial frequencies)

removal of inter-picture redundancy :

coding of difference with an interpolated picture (moving vectors)

high frequent spatial frequencies quantized with lower resolution

than low ones(remove irrelevancy)

Zig-zag scan and VLC (remove redundancy)



10


Result 4:2:2 CCIR 601 resolution : 166 Mbps

(=25images/sec *576lines* 720pixels* 2(lum & chrom) *8bits)

3-4 Mbps (mean) in MPEG2

4:2:0 SIF resolution : 30 Mbps

(=25 images/sec *288 lines *352pixels* 1.5(lum & chrom)*8bits)

1.2 Mbps (CBR) in video CD (MPEG1)



11


Spatial redundancy reduction (DCT example)

158 0 -1 0 0 0 0 0

-1 -1 0 0 0 0 0 0

-1 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0

139 144 149 153 155 155 155 155

144 151 153 156 159 156 156 156

150 155 160 163 158 156 156 156

159 161 162 160 160 159 159 159

159 160 161 162 162 155 155 155

161 161 161 161 160 157 157 157

162 162 161 163 162 157 157 157

162 162 161 161 163 158 158 158

158 0 -1 -1 -1 -1 EOB

zig-zag scan

1260 -1 -12 -5 2 -2 -3 1

-23 -17 -6 -3 -3 0 0 -1

-11 -9 -2 2 0 -1 -1 0

-7 -2 0 1 1 0 0 0

-1 -1 1 2 0 -1 1 1

2 0 2 0 -1 1 1 -1

-1 0 0 -1 0 2 1 -1

-3 2 -4 -2 2 1 -1 0

DCT

Quantisation



12


Temporal redundancy reduction

B

5

Bi-direction al p rediction

I : Intra-co ded pictureP: Pred icted pictureB: Bi-directionally interpolated picture

4

B

rde r ofrese nta tion

rde r ofransm ission

BI P

0 3

B P

1 2 6

B

Prediction

I B P B

nc reas e ofom pressionate

0 1 2 3 4

7

B P B

5 9

B I P

8

P B B P I B

86 7 9



13


Model of a possible encoder

Buffer

Data

Regulator

Motionestimation

Motionvectors

andmodes

Formatconversionand picturereorder

putictures

Picturestore and

prediction

1/DCT

DCT VLC

1/Q

Q Multi-plex



14


MPEG1 en MPEG2 video features MPEG1 sequential picture

resolution : SIF format 288(240)*356*24,25 or 30 Hz

MPEG2 sequential or interlaced

various levels : low level (SIF: 288*356), main level (CCIR601: 576 *720), high 1440 level (HDTV: 1152*1440), high level (EQTV:

1152*1920)

various profiles (toolboxes) : simple profile (No B picture), main profile

(=MPEG1+interlaced), SNR scalable profile (allows graceful degradation

(noise improvement at same resolution), spatial scalable profile

(hierarchical coding : improvement at higher resolution), high profile.


15/39


15

MPEG-4 AVC or H264

Defined about 10 years after MPEG-2

Similar structure used as for MPEG-2, but with some improvements

And two additional functional blocks: Intra-frame prediction: exploit the redundancy between neighboring blocks within a

frame. A block is predicted using previously decoded neighboring blocks.

Deblocking filter: filter applied at the block edges (except picture boundaries) in order to

improved perceptual quality

Cost/benefits Higher complexity

Compression efficiency: factor 2

More than compression

Network adaptation layer defining packets adapted for each transport network

Uses HD-DVD (MPEG2 & H264) & HD-camcorder

Video services over IP networks

Mobile television


16/39


16

Agenda



The MPEG model


context


17/39

December, 20, 2006AV Compression / Alain Bouffioux17

Synchronisation

Synchronisation in the multimedia context

refers to the mechanism that ensures a temporal

consistent presentation of the audio-visual

information to the user


18/39


Intramedia synchronisation

T between capture & presentation = ConstantSame clock frequency & Data on time

Need for corresponding tools

Network

T1T2 =

T1

Capture time

Audiosignal

Encoder

T = Constant

Decoder

Presentation time

Audiosignal


19/39


Intermedia synchronisation

T_Audio = T_Video

Sampled at the same time Presented at the same time)Possible tools : common time base and presentation control (media

synchronisation with the common time base)

Ex.: Lip_sync (requirement: |delay_difference| < 80msec)

Network

T_audio = Constant

T_video = Constant =Capture time

ideoignal

udioignal

Encoder

Capture time

T_AudioPresentation time

Decoder

Videosignal

Audiosignal

Presentation time


20/39


Recovery of clock in CBR

CBR = Constant Bit Rate

if the clock to recover is synchronous with transport clock Recovery ofclock but not of common time base

Remark : possibility to slave stream from DSM (Digital Storage Media) to

local clock

Filter

BR stream

Phaseerror

Filling level

50%

ime Information carriedy each sample

VCO

Processing

Recoveredclock

time


21/39


Recovery of clock and time base in VBR

VBR = Variable Bit Rate

Need for insertion of time stamps (OUTPUT TIME)Output time stamp says for example : It is now 16h25

Receiver adjusts its own horloge to the received time stamp

Recovery of clock & of common time base

Time information carriedonly by time stamps

Time stampextraction

Time counter

Recovery of clock & time

First time stamp

Others

lock

Data stream

Data stream

Time stamps

Counter

Recovered clock

Error Filter VCO

VBR Stream

Counter sample(=Time stamps)

Channel

Time stamps


22/39


Synchronisation with common time base

Insertion of time stamp (=INPUT TIME)Input time stamp says : Input has been sampled at 16h29.Receiver presents the sample at (its input time stamp + maximum encodingand decoding delay).Alternative: transmission of presentation time stamp (input time+delay)

Buffering

Med iaoutp ut

Med ia input

Time c lock(Rec ov ered )

Compa rison of time clockwith sam pled time clock

Samp le "T ime clock "

Ass em ble frame

Time c lock

Timestamp

Proce ss ing

Proce ss ing

Chann el


23/39


Getting data on time

On time Not too late, not too earlyNo buffer over- or underflow

Flow control : not applicable in broadcasting

Common time base and

Definition of a standard target decoder that describes the data

consumption pattern of the receiver.

Remark: Direct MPEG (Microsoft) does not use time information for

clock recovery but relies on flow control


24/39


Streams

Idea of continuity (pipelining)Carry time information for clock recovery

No flow control (allows broadcasting)

The emitter must have a precise knowledge of the receiver data

consumption pattern (explicit in MPEG STD)

Just-in-time

Shorter delay and smaller buffer size than with flow control

Two aspects in synchronisation :

Clock recovery & timing control (model & buffering)


25/39


Requirement on for stream transport

Data information BER (Bit Error Rate) requirement

No repetition of frame possible FEC (Forward Error Correction)

Time information No jitter


26/39


Agenda

About video formats

Video compression


The MPEG model

The MPEG model and its situation in a communication/storagecontext


27/39


What is MPEG ? (1/2)

Moving Picture Expert Group

International standard (ISO/IEC)

Interoperability & economy of scale

Various standard developed/under development

Not only audio/video compression

MPEG-1/2/4 - Compression of audio and video and multiplexing in

a single stream

Definition of the interface not of the codecs

room for improvement

MPEG-1 : until 1.5 Mbps, for DSM

Progressive picture, stereo (Dolby surround)


28/39


What is MPEG ? (2/2)

MPEG-2 : Various bit rates (CBR & VBR)

Program stream for DSM, transport stream for network

Interlaced picture, 5.1 audio channels

Definition of various video levels (e.g. CCIR601 resolution: 4-9 Mbps,

HDTV:15-25 Mbps) and profiles

MPEG-3 : Cancelled, integrated in MPEG-2(Initially : for HDTV)

MPEG-4 : standard for audio, video and graphics in interactive 2D

and 3D multimedia communication.

(Initially : low bit rate for real-time personal communication)

MPEG-7 : Multimedia contents description interface

MPEG-21 : Focus on multimedia distribution and on DRM aspects.


29/39


The MPEG model (1/2)

Audiodecoder

Audiosignal

Videosignal

Presented signals

Multiplexer

Videodecoder

Captured signals

Audioencoder

Videoencoder

udioignal

ideoignal Digital storage medium

orNetwork

Transmissionchannel

Demulti-plexer


30/39


The MPEG model (2/2)

Compression of audio & video and multiplexing in a single stream

Guarantees intramedia and intermedia synchronisation.

MPEG defines an interface bitstream syntax

timing of the bitstream STD specifying timing requirement (ideal

model)

Consequences: Decoder should compensate deviations from STD

Network should correct jitter introduced by the channel (RTD-LJ)

MPEG stream must be adapted to transmission channel formatting,error correction, channel coding (b.v.video-CD)


31/39


Components of the MPEG standard

The MPEG standard is composed of 3 main parts : Audio : Specifies the compression of audio signals

Video : Specifies the compression of video signals

System : specifies how the compressed audio and video signals are

combined in the multiplexed stream (program stream or transport

stream).

Each part specifies : The bitstream syntax

The timing requirement and the related information (bit rate, buffer

needs)


32/39


Synchronisation Mechanism (1/2)

Syste m decoderand

demu ltiplexe r

Mu ltiplexerand systemencoder

Audiodecoder

Vide odecoder

Comp ar isonPTS and STCandpresenta tion

Vide ooutpu t

Comp ar isonPTS and STCandpresenta tion

Audioou tpu t

Extr actionof PCR(SCR )

STC

Audioencoder

Vide oencoder

Assemblepictu res,Samp le STCfor PTS

ideopu t

Assembleaudio fra me s,Samp le STCfor PTS

ud iopu t

Samp le STCfor PC R(SCR )

STC

Transm issionchannel


33/39


Synchronisation Mechanism (2/2)

PCR for TS & SCR for PS (but same concept)

Clock & time base recovery: Time-stamping at OUTPUT (PCR included in

TS multiplex, SCR in pack header)

Audio & video clock locked to STC easy recovery (see next slide)

Synchronisation of audio & video to common time base (Time stamping at

Input)

STD is defined (because of the absence of flow control)

streams are such that STD buffers never over- or underflow

In TS, many program in a single stream but unique clock per program.

Time information No Jitter requirement for transport


34/39


Clock recovery in receiver

VCOAudio clock

divider

Video clockdivider Video

clock

Audioclock

Audiooutput

STC

Comparis onPTS and STCand

PresentationDecodedaudio

PTS

STC(Counter)

CR

STC

ErrorLow Pass Filter(Integrator)

Load firs t PCR


35/39


MPEG-2 program & transport streams

Program streams: Relatively error free environment

program stream packet may have variable and great length

Single time base

Transport streams: environment where errors are likely

many programs (independent time base)

Transport stream packet : fixed, 188 bytes

Contains tables


36/39


Channelencode

Channeldecode

Bit-stream

Sourcedecode

formationink

Digitaloutput

Format Decrypt

Sourceencode

Format

formationource

Digitalinput

Encrypt

Synchro-nisation

Digitalwaveform

Otherdestination

Demodu-lateDemulti-plex Multipleaccess

Modu-late

Multi-plex

Channelbits

Othersources

Multipleaccess

Channel

MPEG in a communication context (1)

Typical communication system


37/39


MPEG in a communication context (2)

MPEG : Source coding only (bit rate reduction) + multiplexing

The MPEG stream must be adapted to the channel in what concern its physical

characteristics and in order to get the required QoS (Quality of Service) &

Security Encryption

Channel coding (forward error correction, interleaving, modulation codes)

multiplexing & formatting modulation (frequency allocation)

multiple access method

Some channels : CD/DVD - satellite - cable - ATM - 1394


38/39


VideoEncoder

PEG2 compression layer

Audioencoder

udio,ideo

ources

ES

(ElementaryStream)

Adap-tationto the

channel

PS

(1 pro-gram)

MPEG2 system layer

PSMulti-plexing

Adap-

tationto thechannel

DVB, DVD ...

Disc

Satellite

TSMulti-plexing

TS(n pro-

grams)

Adap-tationto the

channelCable

TS (Transport Stream)

orPS (Program Stream)

MPEG-2 in a communication context (3)

A simple view of MPEG-2 in the communication context


39/39

39

3. av compression - more about video compression & mpeg - dec06

Documents