audio/video compression more about video compression and mpeg alain bouffioux december, 20, 2006

Audio/Video compressionMore about video compression and MPEG

Alain Bouffioux

December, 20, 2006

December, 20, 2006AV Compression / Alain Bouffioux

2

Agenda

About video formats

Video compression

Audio/Video synchronisation

The MPEG model

The MPEG model and its situation in a communication/storage context


3

Agenda

About video formats

Video compression


The MPEG model



4

Some video formats (1)

Max. component video signal bandwidth: 6 MHz.

CCIR601 (CCIR is now ITU-R): Video sampling frequency: 13.5 MHz for 525 & 625 line standards(Shannon requirement)

Synchronous with line (& image) sampling frequencyFsampling= 864*Fh for 625 line system (50Hz countries) Fsampling= 858*Fh for 525 line system (60Hz countries)

Why synchronous? Points at the same place

RGB format


5


YCbCr formatCb = B-Y, Cr = R-YEye is more sensitive to luminance than to chrominance (lower resolution needed for chrominance)

R ed

B lueG reen

M atrixLP F

Y

C b

C r

M odula torS ubC

+C om posite

V ideo


6


The 4:2:2 format– Y sampling @ 13.5 MHz– C sampling @ 6.75 MHz– 8 bits per pixel– 720 active points per line– 576 lines active lines per image (2 fields) (625 lines)

and 480 active lines (525 lines) – Pixels are not square (e.g. for 480 lines, only 640 active points are needed

- VGA format)– Image size 720*576 or 720*480

The 4:2:0 format– Vertical chrominance resolution reduced by a factor 2

(average on two successive lines)


7


SIF format (Source Intermediate Format)Half the vertical & horizontal resolution of 4:2:0For 50Hz countries:

– Luminance: 360*288– Chrominance: 180*120

CIF format (Common Intermediate Format)– Intermediate format used in videoconferencing

(communication between US & Europe)– resolution: 360*288 – Sampling frequency: 30 Hz

QCIF (Quarter CIF)– Half the vertical & horizontal resolution of CIF.


8

Agenda

About video formats

Video compression


The MPEG model



9

Video compression in MPEG-1&2 (1/6)

Principles– removal of intra-picture redundancy :

Image is decomposed in 8*8 pixels sub-images.Each sub-image contains redundant information DCT transformation (in frequency domain) de-correlates the input signal.( most energy in low spatial frequencies)

– removal of inter-picture redundancy :coding of difference with an interpolated picture (moving vectors)

– high frequent spatial frequencies quantized with lower resolution than low ones(remove irrelevancy)

– Zig-zag scan and VLC (remove redundancy)


10


Result– 4:2:2 CCIR 601 resolution : 166 Mbps

(=25images/sec *576lines* 720pixels* 2(lum & chrom) *8bits) ± 3-4 Mbps (mean) in MPEG2

– 4:2:0 SIF resolution : 30 Mbps (=25 images/sec *288 lines *352pixels* 1.5(lum & chrom) *8bits) ±1.2 Mbps (CBR) in video CD (MPEG1)


11


Spatial redundancy reduction (DCT example)

158 0 -1 0 0 0 0 0 -1 -1 0 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

139 144 149 153 155 155 155 155144 151 153 156 159 156 156 156150 155 160 163 158 156 156 156159 161 162 160 160 159 159 159159 160 161 162 162 155 155 155161 161 161 161 160 157 157 157162 162 161 163 162 157 157 157162 162 161 161 163 158 158 158

158 0 -1 -1 -1 -1 EOBzig-zag scan

1260 -1 -12 -5 2 -2 -3 1 -23 -17 -6 -3 -3 0 0 -1 -11 -9 -2 2 0 -1 -1 0 -7 -2 0 1 1 0 0 0 -1 -1 1 2 0 -1 1 1 2 0 2 0 -1 1 1 -1 -1 0 0 -1 0 2 1 -1 -3 2 -4 -2 2 1 -1 0

DCT

Quantisation


12


Temporal redundancy reduction

B

5

Bi-directional prediction

I : Intra-coded pictureP: Predicted pictureB: Bi-directionally interpolated picture

4

B

Order ofpresentation

Order oftransmission

BI P

0 3

B P

1 2 6

B

Prediction

I B P B

Increase of compressionrate

0 1 2 3 4

7

B P B

5 9

B I P

8

P B B P I B

86 7 9


13


Model of a possible encoder

Buffer

Data

Regulator

Motion estimation

Motionvectorsand modes

Format conversion and picture reorder

Inputpictures

Picturestore andprediction

1/DCT

DCT VLC

1/Q

Q Multi-plex


14


MPEG1 en MPEG2 video features– MPEG1

• sequential picture• resolution : SIF format 288(240)*356*24,25 or 30 Hz

– MPEG2• sequential or interlaced• various levels : low level (SIF: 288*356), main level (CCIR601: 576 *

720), high 1440 level (HDTV: 1152*1440), high level (EQTV: 1152*1920)

• various profiles (toolboxes) : simple profile (No B picture), main profile (=MPEG1+interlaced), SNR scalable profile (allows graceful degradation (noise improvement at same resolution), spatial scalable profile (hierarchical coding : improvement at higher resolution), high profile.


15

MPEG-4 AVC or H264

Defined about 10 years after MPEG-2

Similar structure used as for MPEG-2, but with some improvements

And two additional functional blocks:– Intra-frame prediction: exploit the redundancy between neighboring blocks within a

frame. A block is predicted using previously decoded neighboring blocks.– Deblocking filter: filter applied at the block edges (except picture boundaries) in order to

improved perceptual quality

Cost/benefits– Higher complexity– Compression efficiency: factor 2

More than compressionNetwork adaptation layer defining packets adapted for each transport network

Uses – HD-DVD (MPEG2 & H264) & HD-camcorder– Video services over IP networks – Mobile television


16

Agenda

About video formats

Video compression


The MPEG model



17

Synchronisation

Synchronisation in the multimedia context

refers to the mechanism that ensures a temporal

consistent presentation of the audio-visual

information to the user


18

Intramedia synchronisation

T between capture & presentation = Constant Same clock frequency & Data on time Need for corresponding tools

Network

T1T2 =

T1

Capture time

Audio signal Encoder

T = Constant

Decoder

Presentation time

Audio signal


19

Intermedia synchronisation

T_Audio = T_Video Sampled at the same time Presented at the same time) Possible tools : common time base and presentation control (media

synchronisation with the common time base)

Ex.: Lip_sync (requirement: |delay_difference| < 80msec)

Network

T_audio = Constant

T_video = Constant = Capture time

Video signal

Audio signal

Encoder

Capture time

T_Audio Presentation time

Decoder

Video signal

Audio signal

Presentation time


20

Recovery of clock in CBRCBR = Constant Bit Rate

if the clock to recover is synchronous with transport clock Recovery of clock but not of common time base

Remark : possibility to slave stream from DSM (Digital Storage Media) to local clock

Filter

CBR stream

Phaseerror

Filling level

50%

Time Information carriedby each sample

VCO

Processing

Recoveredclock

time


21

Recovery of clock and time base in VBRVBR = Variable Bit Rate

Need for insertion of time stamps (OUTPUT TIME)Output time stamp says for example : “It is now 16h25”Receiver adjusts its own horloge to the received time stamp

Recovery of clock & of common time baseTime information carriedonly by time stamps

Time stamp extraction

Time counter

Recovery of clock & time

First time stamp

Others

Clock

Data stream

Data stream

Time stamps

Counter

Recovered clock

Error Filter VCO

VBR Stream

Counter sample(=Time stamps)

Channel

Time stamps


22

Synchronisation with common time base

Insertion of time stamp (=INPUT TIME)Input time stamp says : “Input has been sampled at 16h29”.Receiver presents the sample at (its input time stamp + maximum encoding and decoding delay).Alternative: transmission of presentation time stamp (input time+delay)

Buffering

Mediaoutput

Media input

Time clock(Recovered)

Comparison of time clockwith sampled time clock

Sample "Time clock"Assemble frame

Time clock

Timestamp

Processing

Processing

Channel


23

Getting data on time

“On time” Not too late, not too earlyNo buffer over- or underflow

Flow control : not applicable in broadcasting

Common time base and Definition of a standard target decoder that describes the data consumption pattern of the receiver.

Remark: Direct MPEG (Microsoft) does not use time information for clock recovery but relies on flow control


24

Streams

Idea of continuity (pipelining)

Carry time information for clock recovery

No flow control (allows broadcasting)The emitter must have a precise knowledge of the receiver data consumption pattern (explicit in MPEG STD)

Just-in-timeShorter delay and smaller buffer size than with flow control

Two aspects in synchronisation :Clock recovery & timing control (model & buffering)


25

Requirement on for stream transport

Data information BER (Bit Error Rate) requirementNo repetition of frame possible FEC (Forward Error Correction)

Time information No jitter


26

Agenda

About video formats

Video compression


The MPEG model



27

What is MPEG ? (1/2)

Moving Picture Expert Group

International standard (ISO/IEC) Interoperability & economy of scale

Various standard developed/under developmentNot only audio/video compression

MPEG-1/2/4 - Compression of audio and video and multiplexing in a single stream

Definition of the interface not of the codecs room for improvement

MPEG-1 : until 1.5 Mbps, for DSMProgressive picture, stereo (Dolby surround)


28

What is MPEG ? (2/2)

MPEG-2 : Various bit rates (CBR & VBR)Program stream for DSM, transport stream for networkInterlaced picture, 5.1 audio channels Definition of various video levels (e.g. CCIR601 resolution: 4-9 Mbps, HDTV:15-25 Mbps) and profiles

MPEG-3 : Cancelled, integrated in MPEG-2(Initially : for HDTV)

MPEG-4 : standard for audio, video and graphics in interactive 2D and 3D multimedia communication. (Initially : low bit rate for real-time personal communication)

MPEG-7 : Multimedia contents description interface

MPEG-21 : Focus on multimedia distribution and on DRM aspects.


29

The MPEG model (1/2)

Audiodecoder

Audio signal

Videosignal

Presented signals

Multiplexer

Videodecoder

Captured signals

Audioencoder

Videoencoder

Audio signal

Videosignal Digital storage medium

orNetwork

Transmission channel

Demulti-plexer


30

The MPEG model (2/2)

Compression of audio & video and multiplexing in a single stream

Guarantees intramedia and intermedia synchronisation.

MPEG defines an interface– bitstream syntax– timing of the bitstream STD specifying timing requirement (ideal

model)

Consequences:– Decoder should compensate deviations from STD– Network should correct jitter introduced by the channel (RTD-LJ)

MPEG stream must be adapted to transmission channel formatting, error correction, channel coding (b.v.video-CD)


31

Components of the MPEG standard

The MPEG standard is composed of 3 main parts :– Audio : Specifies the compression of audio signals– Video : Specifies the compression of video signals– System : specifies how the compressed audio and video signals are

combined in the multiplexed stream (program stream or transport stream).

Each part specifies :– The bitstream syntax – The timing requirement and the related information (bit rate, buffer

needs)


32

Synchronisation Mechanism (1/2)

System decoderanddemultiplexer

Multiplexer and systemencoder

Audiodecoder

Videodecoder

ComparisonPTS and STCand presentation

Videooutput

ComparisonPTS and STCand presentation

Audiooutput

Extractionof PCR (SCR)

STC

Audioencoder

Videoencoder

Assemble pictures,Sample STC for PTS

Videoinput

Assembleaudio frames,Sample STCfor PTS

Audioinput

Sample STCfor PCR(SCR)

STC

Transmission channel


33

Synchronisation Mechanism (2/2)

PCR for TS & SCR for PS (but same concept)

Clock & time base recovery: Time-stamping at OUTPUT (PCR included in TS multiplex, SCR in pack header)

Audio & video clock locked to STC easy recovery (see next slide)

Synchronisation of audio & video to common time base (Time stamping at Input)

STD is defined (because of the absence of flow control)streams are such that STD buffers never over- or underflow

In TS, many program in a single stream but unique clock per program.

Time information “No Jitter” requirement for transport


34

Clock recovery in receiver

VCO Audio clockdivider

Video clockdivider Video

clock

Audioclock

Audiooutput

STC

ComparisonPTS and STCandPresentationDecoded

audio

PTS

STC(Counter)

PCR

STC

Error Low Pass Filter(Integrator)

Load first PCR


35

MPEG-2 program & transport streams

Program streams:– Relatively error free environment– program stream packet may have variable and great length– Single time base

Transport streams:– environment where errors are likely– many programs (independent time base)– Transport stream packet : fixed, 188 bytes– Contains tables


36

Channelencode

Channeldecode

Bit-stream

Sourcedecode

Informationsink

Digitaloutput

Format Decrypt

SourceencodeFormat

Informationsource

Digitalinput

Encrypt

Synchro-nisation

Digitalwaveform

Otherdestination

Demodu-late

Demulti-plex

Multipleaccess

Modu-late

Multi-plex

Channelbits

Othersources

Multipleaccess

Channel

MPEG in a communication context (1)

“Typical” communication system


37

MPEG in a communication context (2)

MPEG : Source coding only (bit rate reduction) + multiplexing

The MPEG stream must be adapted to the channel in what concern its physical characteristics and in order to get the required QoS (Quality of Service) & Security

– Encryption – Channel coding (forward error correction, interleaving, modulation codes)– multiplexing & formatting– modulation (frequency allocation)– multiple access method

Some channels : CD/DVD - satellite - cable - ATM - 1394


38

VideoEncoder

MPEG2 compression layer

Audioencoder

Audio,videosources

ES(ElementaryStream)

Adap-tationto thechannel

PS(1 pro-gram)

MPEG2 system layer

PSMulti-plexing

Adap-tationto thechannel

DVB, DVD ...

Disc

Satellite

TSMulti-plexing

TS(n pro-grams)

Adap-tationto thechannel Cable

TS (Transport Stream)orPS (Program Stream)

MPEG-2 in a communication context (3) A simple view of MPEG-2 in the communication context


39

audio/video compression more about video compression and mpeg alain bouffioux december, 20, 2006

Documents