cs 414 - spring 2012 cs 414 – multimedia systems design lecture 12 – mpeg-2/ mpeg-4 (part 6)...

CS 414 - Spring 2012

CS 414 – Multimedia Systems Design Lecture 12 – MPEG-2/MPEG-4 (Part 6)

Klara Nahrstedt

Spring 2012


Administrative MP1

Outline MPEG-2/MPEG-4 Reading:

Media Coding book, Section 7.7.2 – 7.7.5 http://www.itu.int/ITU-D/tech/digital-broadcasting/kiev/References/mpe

g-4.html http://en.wikipedia.org/wiki/H.264

Available software Xvid – free software http://www.bing.com/videos/watch/video/xvid-free-download-mpeg-4-vi

deo-codec-for-pc


MPEG-2 Standard Extension MPEG-1 was optimized for CD-ROM and

apps for 1.5 Mbps (video strictly non-interleaved)

MPEG-2 adds to MPEG-1: More aspect ratios: 4:2:2, 4:4:4Progressive and interlaced frame codingFour scalable modes

spatial scalability, data partitioning, SNR scalability, temporal scalability


MPEG-1/MPEG-2

Pixel-based representations of content takes place at the encoder

Lack support for content manipulatione.g., remove a date stamp from a video turn off “current score” visual in a live game

Need support manipulation and interaction if the video is aware of its own content


MPEG-4 Compression


Interact With Visual Content


Original MPEG-4

Conceived in 1992 to address very low bit rate audio and video (64 Kbps)

Required quantum leaps in compressionbeyond statistical- and DCT-based techniquescommittee felt it was possible within 5 years

Quantum leap did not happen


The “New” MPEG-4 Support object-based features for content

Enable dynamic rendering of contentdefer composition until decoding

Support convergence among digital video, synthetic environments, and the Internet


MPEG-4 Components Systems – defines architecture, multiplexing structure, syntax Video – defines video coding algorithms for animation of synthetic and

natural hybrid video (Synthetic/Natural Hybrid Coding) Audio – defines audio/speech coding, Synthetic/Natural Hybrid Coding

such as MIDI and text-to-speech synthesis integration Conformance Testing – defines compliance requirements for MPEG-4

bitstream and decoders Technical report DSM-CC Multimedia Integration Framework – defines Digital Storage

Media – Command&Control Multimedia Integration Format; specifies merging of broadcast, interactive and conversational multimedia for set-top-boxes and mobile stations


MPEG-4 Example

ISO N3536 MPEG4 CS 414 - Spring 2012

MPEG-4 Example

Daras, P. MPEG-4 Authoring Tool, J. Applied Signal Processing, 9, 1-18, 2003


Interactive Drama

http://www.interactivestory.netCS 414 - Spring 2012

MPEG-4 Characteristics and Applications


Example of MPEG-4 Scene

Source: http://mpeg.chiariglione.org/standards/mpeg-4/mpeg-4.htm

Media Objects

An object is called a media objectreal and synthetic images; analog and

synthetic audio; animated faces; interaction

Compose media objects into a hierarchical representation form compound, dynamic scenes


Logical Structure of Scene


Example of Composed Scene


Sprite ObjectPlayer Object

Composed Scene

Video Syntax Structure


New MPEG-4 Aspect: Object-based layered syntactic structure

Content-based Interactivity Achieves different qualities for different objects

with a fine granularity in spatial resolution, temporal resolution and decoding complexity

Needs coding of video objects with arbitrary shapes

Scalability Spatial and temporal scalability Need more than one layer of information (base and

enhancement layers)


Examples of Base and Enhancement Layers


Coding of Objects Each VOP corresponds to an entity that after

being coded is added to the bit stream Encoder sends together with VOP

Composition information where and when each VOP is to be displayed

Users are allowed to change the composition of the entire scene displayed by interacting with the composition information


Spatial Scalability


VOP which is temporally coincident with I-VOP in the base layer, is encoded as P-VOP in the enhancement layer. VOP which is temporally coincident with P-VOP in the base layer is encoded as B-VOP in the enhancement layer.

Examples of Base and Enhancement Layers


Temporal Scalability


High Level Codec for Generalized Scalability


Composition (cont.)

Encode objects in separate channelsencode using most efficient mechanism transmit each object in a separate stream

Composition takes place at the decoder,rather than at the encoder requires a binary scene description (BIFS)

BIFS is low-level language for describing:hierarchical, spatial, and temporal relations


MPEG-4 Part 11 – Scene Description BIFS – Binary Format for Scenes

Coded representation of the spatio-temporal positioning of audio-visual objects as well as their behavior in response to interactions

Coded representation of synthetic 2D and 3D objects that can be manifested audibly and/or visibly

BIFS – MPEG-4 scene description protocol to Compose MPEG-4 objects Describe interaction about MPEG-4 objects Animate MPEG-4 objects

BIFS Language Framework – XMT (textual representation of multimedia content using XML) Accommodates SMIL, W3C scalable vector graphics and VRML (now

X3D)


MPEG-4 Rendering

ISO N3536 MPEG4 CS 414 - Spring 2012

Major Components on MPEG-4 Terminal

http://mpeg.chiariglione.org/standards/mpeg-4/mpeg-4.htmCS 414 - Spring 2012

Integration and Synchronization of Multiple Streams


Source: http://mpeg.chiariglione.org/technologies/mpeg-4/mp04-bifs/index.htm

Synchronization

Global timeline (high-resolution units)e.g., 600 units/sec

Each continuous track specifies relatione.g., if a video is 30 fps, then a frame should

be displayed every 33 ms.

Others specify start/end time


Data Recovery


• Reversible Variable Length Codes (RVLC) : variable length code words are designed such that they can be read both in forward and reverse directions

QP – quantization parameterHEC – header extension codeMP – macro-block

MPEG-4 parts MPEG-4 part 2

Includes Advanced Simple Profile, used by codecs such as Quicktime 6

MPEG-4 part 10 MPEG-4 AVC/H.264 also called Advanced

Video Coding Used by coders such as Quicktime 7Used by high-definition video media like Blu-

ray DiscCS 414 - Spring 2012

MPEG-4 System Layer Model

http://mpeg.chiariglione.org/standards/mpeg-4/mpeg-4.htm

SL – sync layerDMIF – Delivery Multimedia Integration Framework

MPEG-4 File with Media Data

http://mpeg.chiariglione.org/standards/mpeg-4/mpeg-4.htm

OD – Object DescriptorBIFS – Binary Format for ScenesMdat – media data atoms

Interaction as Objects

Change colors of objects Toggle visibility of objects Navigate to different content sections Select from multiple camera views

change current camera angle Standardizes content and interaction

e.g., broadcast HDTV and stored DVD


Hierarchical Model

Each MPEG-4 movie composed of trackseach track composed of media elements (one

reserved for BIFS information)each media element is an objecteach object is a audio, video, sprite, etc.

Each object specifies its:spatial information relative to a parent temporal information relative to global timeline


Conclusion A lot of MPEG-4 examples with interactive capabilities Content-based Interactivity

Scalability Sprite Coding

Improved Compression Efficiency (Improved Quantization)

Universal Accessibility re-synchronization data recovery error concealment


cs 414 - spring 2012 cs 414 – multimedia systems design lecture 12 – mpeg-2/ mpeg-4 (part 6)...

Documents

outline mpeg

new mpeg

example of mpeg

original mpeg

compression cs

standard extension mpeg

exampleiso n3536 mpeg4

digital video