mpeg-4 avc / h.264 coding standard avc / h.264 ...vca.ele.tue.nl/sites/default/files/5lse0 mod 11...
TRANSCRIPT
1
1
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
Multimedia Video Coding & Architectures
(5LSE0), Module 11
MPEG-4 AVC / H.264 Coding
Peter H.N. de With
2
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 (MPEG-4 AVC) Standard / Contents
Part 1: General H.264 system properties and outline∗ Introduction to standard and history∗ Compression capabilities & complexity (vs. MPEG-2) ∗ Block diagram and some key properties
Part 2: Video Coding Layer∗ Video data partitioning and macroblocks∗ Motion Compensation and Prediction Modes∗ Integer Transform and Deblocking Filter
Part 3: VLC Coding Techniques and Compr. Performance∗ VLC Techniques: CAVLC and CABAC∗ Performance comparison under various conditions∗ Implementation aspects
3
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
MPEG-4 AVC / H.264 Coding Standard
Part 1: Introduction and overview
Peter H.N. de WithEindhoven Univ. of Technology, The Netherlands
TU Eindhoven
Faculty EE, SPS-VCA
Flux 5.092
PO Box 513,
5600MB Eindhoven,
The Netherlands
4
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 - Preliminaries
∗ Advanced Video Coding definition• Algorithm, Standard • Profiles and Levels for various applications
∗ Rapid acceptance of standard, because … – Emerged from two communities simultaneously: both
IEC MPEG and ITU H.26L standards committees
– Joint Video Team shared development in 1999-2003
– Offers significant improvement in compression
– One major application immediately avalaible: the next generation DVD (2006): Blu-ray Disc (BD) and HD-DVD
2
5
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 Intro - Applications
∗ Entertainment Video (1-8+ Mbit/s, higher latency)– Broadcast / Satellite / Cable / DVD / VoD / FS-VDSL / …– DVB / ATSC / SCTE, DVD Forum, DSL Forum
∗ Conversational Services (usually < 1 Mbit/s, low latency)– H.320 Conferencing /Conversational - circuit-switched– 3GPP Conferencing /Conversational H.324/M - circuit-switched– H.323 Conversational Internet / best effort IP/RTP - packet-switch‘d– 3GPP Conversational IP/RTP/SIP - packet-switch‘d
∗ Streaming Services (typ. lower bit rate, higher latency) – 3GPP Streaming IP/RTP/RTSP– Streaming IP/RTP/RTSP (without TCP fallback)
∗ Other Services– 3GPP Multimedia Messaging Services
6
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 Intro –
Relation to other standards
∗ Identical specifications have been approved in ITU-T VCEG and
ISO/IEC MPEG
∗ In ITU-T / VCEG this is a new & separate standard
– ITU-T Recommendation H.264
– ITU-T Systems (H.32x) will be modified to support it
∗ In ISO/IEC MPEG this is a new „part“ in the MPEG-4 suite
– Separate codec design from prior MPEG-4 visual
– New Part 10 called Advanced Video Coding (AVC), similar to „AAC“ position in
MPEG-2 as separate codec
– MPEG-4 Systems / File format has been modified to support it
– H.222.0 | MPEG-2 Systems also modified to support it
∗ IETF finalizing RTP payload packetization
7
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 Overview /
Evolution Video Codec Technology..1990 2000
ITU-T
ISO / IEC
Same as High level of commonality
CE Industry Consortium
Proprietary
SMPTE
JVT
8
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
Evolution of AVC / H.264
De-quantizer / Inv. Transform
Motion-Compensated
Predictor
0
Intra/Inter
CoderControl
Decoder
MotionEstimator
Transform/Quantizer-
16*1616*88*16
8*88*44*8
4*4
intra, predictive and bi-predictive coding using
multiple reference
pictures
4*4 Integer Transform
UVLC,CABAC
AVC (H.264)
MB
1, 1/2,1/4, 1/8 pel
QuantizedTransf. coeff’s
EntropyCoding
Motiondata
Controldata
3
9
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / Benefits of coding with AVC
∗ AVC (H.264 / MPEG-4 part 10)
– About twice as efficient than MPEG-2
• Same content at lower datarate
• Higher resolution content at same datarate
– Architecture comparable to MPEG-2
• More sophisticated tools
• Important differences in many details (4x4 instead of 8x8 DCT)
– Truly open industry standard
– Different profiles allow for usage from mobile applications
to full High Definition TV
10
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
Coding Efficiency (MPEG-2 vs. AVC)Average bitrate at SD resolution (720 * 576/480)
6
5
4
3
2
1
Mb/s
1995 2000 2005
AVC
MPEG-2Improvements due to:• improved encoding algorithm
• technology progress (further integration)• use of variable bitrate instead of constant bitrate
• improved pre-processing (e.g noise reduction)
AVC about 2 x more efficientFor HD possibly less (~ 1.5)
11
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / Detailed view on bit-rate efficiency
MPEG-2
H.263
MPEG-4
H.26L H.264
12
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
Complexity of MPEG-2 vs. AVCIC area
1995 2000 2005
AVCMPEG-2
New IC technologies
reduces IC area
Reduced to about ¼ of initial value
AVC about 4 x more complex than MPEG-2
Required IC area now
about the same as for
MPEG-2 at introduction
4
13
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 New Features∗ Multi-mode, multi-refer. Motion Compensation
∗ Motion vector can point out of image border
∗ 1/4-, 1/8-pixel motion vector precision
∗ B-frame prediction weighting
∗ 4×4 integer transform
∗ Multi-mode intra-prediction
∗ In-loop de-blocking filter
∗ UVLC (Uniform Variable Length Coding)
∗ NAL (Network Abstraction Layer)
∗ SP-slices
14
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 Profiles and Levels – (1)
∗ Many standards contain different configurations of
capabilities – often based in “profiles” & “levels”
– Profile: usually a set of algorithmic features, the so-called
subset of the syntax
– Level: usually a degree of capability, e.g. resolution, speed,
thus the limitations to coding parameters
∗ H.264 / AVC has three profiles
– Baseline - lower capability plus error resilience
– Main - high compression quality
– Extended - added features for efficient streaming
15
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 Profiles and Levels – (2)
∗ Profiles applications: Baseline, Main, and Extended (X)
– Baseline: Progressive low-rate video,
Videoconferencing & Wireless communication
– Main: especially for Broadcast applications, Video Recording
– Extended: Mobile networking
∗ Baseline profile is the minimum implementation
– No use of: CABAC, 1/8 MC, B-frame, SP-slices
∗ 15 Levels
– Resolution, capability, bit rate, buffer, reference #
– Built to match popular int. production and transmission formats
– Ranges from QCIF to D-Cinema
16
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 Layer Structure∗ Layer structure has extended networking capabilities
– Extended interfacing to many standards
– Good concept with Network Abstraction Layer (NAL)
5
17
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC Video Data
Part 2 Video Coding Layer:
A. Data partitioning & Macroblocks
B. Motion Compens. & Prediction Modes
18
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 High-level Video summary
∗ Video coding layer is based on hybrid video coding
– Similar spirit as other MPEG standards, like DCT + Motion compensation
– But with important differences
∗ New key aspects are
– Enhanced motion compensation
– Small blocks and partly variable block size for transform coding
– Improved deblocking filter
– Enhanced entropy coding
∗ Performance: substantial bit-rate savings relative to other
standards for the same quality
– On top of the above key features: smart control of those features
19
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Picture partitioning
∗ Picture divided in slices and MBs
∗ Slices
– Picture has 1 or more slices
– Slices are self-contained
– Slices are a sequence of macroblocks
∗ Macroblocks
– Basic syntax & processing unit
– Contains 16x16 Y (luma) samples and
2x ( 8x8 ) Color (chroma) samples
– Macroblocks within a slice depend on
each other
– Macroblocks can be further partitioned
20
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 –
Flexible Macroblock Ordering
∗ Slice Group
– Pattern of macroblocks defined by a macroblock allocation map
– Slice group may contain 1 to several slices
∗ Macroblock allocation map types
– Interleaved slices
– Dispersed macroblock allocation
– Explicitly assign a slice group to each macroblock location in raster scan order
– One or more „foreground“ slice groups and „leftover“ slice group
∗ Remember: Picture quality …!
6
21
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264-Basic MB-Based Coding Structure
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks
16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
Scheme supports MB
field-based processing
for interlacing also, like
in MPEG-2
22
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 /AVC - Data Encoding Structure
23
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC - Motion Compensation - (1)
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks
16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
Various block sizes and shapes
8x8
0
4x8
0 10 1
2 3
4x48x4
1
08x8
Types
0
16x16
0 1
8x16
MBTypes
8x8
0 1
2 3
16x8
1
0
24
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
Various block sizes and shapes
8x8
0
4x8
0 10 1
2 3
4x48x4
1
08x8
Types
0
16x16
0 1
8x16
MBTypes
8x8
0 1
2 3
16x8
1
0
H.264 / AVC –
Results Motion Compensation - (2)
7
25
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 –
Quarter-sample Y interpolation
∗ Half-sample positions are
obtained by applying a 6-tap
filter with tap values (1, -5,
20,20, -5, 1)
∗ Quarter-sample positions are
obtained by averaging
samples at integer and half-
sample positions
26
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC - Multiple Reference Frames
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks
16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
Multiple Reference Frames for
Motion Compensation
27
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Multiple Reference Frames &
Generalized Bi-Predictive Frames
1. Extend motion vectors by
reference picture index D
2. Provide reference pictures at
the decoder side
3. In case of bi-predictive
pictures: decode 2 sets of
motion parameters
Can jointly exploit scene cuts, aliasing, uncovered
background and other effects with one approach!
28
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 –
New types Temporal Referencing
∗ Known dependencies (MPEG-1 and MPEG-2, etc.)
∗ New types of dependencies
– Referencing order and display order are decoupled
– Referencing ability and picture type are decoupled
8
29
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Weighted Prediction
∗ In addition to shifting in spatial position, and selecting from
multiple ref. pictures, each region‘s prediction values can be
– Multiplied by a weight
– Given an additive offset
∗ Some key issues
– Improved efficiency for B coding, e.g.,
• Accelerating motion,
• Multiple non-reference B temporally between reference pictures
– Excels at representation of fades, such as fade-in, fade-out, cross-fades
between two scenes
∗ Encoder can apply this to both P and B prediction types
30
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 /AVC B-frame Prediction Weighting
∗ Playback order: I0 B1 B2 B3 P4 B5 B6 ……...
∗ Bitstream order: I0 P4 B1 B3 B2 P8 B5 ……...
I0 B1 B2 B3 P4 B5 B6
Time
31
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Spatial Prediction
using surrounding „available“ samples ∗ Available samples are …
– Previously reconstructed within the same slice at the decoder
– Inside the same slice
∗ Luma intra prediction either
– Single prediction for entire 16x16 macroblock
• 4 modes (vertical, horizontal, DC, planar)
– 16 individual predictions of 4x4 blocks
• 9 modes (DC, 8-directional)
∗ Chroma intra prediction
– Single prediction type for both 8x8 regions
• 4 modes (vertical, horizontal, DC,planar)
32
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC - Intra-prediction Modes
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks
16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
Directional spatial prediction
(9 types for luma, 1 chroma)
• e.g., Mode 3:
diagonal down/right prediction
a, f, k, p are predicted by
(A + 2Q + I + 2) >> 2
Q A B C D E F G H
I a b c d
J e f g h
K i j k l
L m n o p
M
N
O
P
1
2
3456
7
8
2- DC
0
9
33
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 –
16x16 Intra Prediction Directions ∗ Mode 0 – Vertical and Mode 1 - Horizontal
34
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 –
4x4 Intra Prediction Directions -(1)
35
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 –
4x4 Intra Prediction Directions -(2)
36
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC Part 2 Video Coding Layer:
Transform Coding
C. Integer Transform,
Deblocking Filter and
Performance comparison
10
37
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC - 4×4 Integer Transform – (1)
EntropyCoding
Scaling & Inv. Transform
Motion-Compensation
ControlData
Quant.Transf. coeffs
MotionData
Intra/Inter
CoderControl
Decoder
MotionEstimation
Transform/Scal./Quant.-
InputVideoSignal
Split intoMacroblocks
16x16 pixels
Intra-frame Prediction
De-blockingFilter
OutputVideoSignal
4x4 Block Integer Transform
Main Profile: Adaptive Block Size
Transform (8x4,4x8,8x8)
Repeated transform of DC coeffs
for 8x8 chroma and 16x16 Intra
luma blocks
1 1 1 1
2 1 1 2
1 1 1 1
1 2 2 1
− − = − −
− −
H
38
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Integer Transform –(2)
∗ Separable transform of a block B4x4 of size 4x4
∗ Th , Tv : horizontal and vertical transform matrix
∗ 4x4 transform matrix:
– Easy implementation
– Different norms for even and odd rows of the matrix
39
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Deblocking filter
∗ Improves subjective visual & objective quality of decoded picture
∗ Significantly superior to post filtering
∗ Filtering affects the edges of the 4x4 block structure
∗ Highly content-adaptive filtering procedure mainly removes
blocking artifacts and only scarcely blurs the visual content
– At the slice level, the global filtering strength can be adjusted to the individual
characteristics of the video sequence
– At the edge level, filtering strength is made dependent on inter/intra, motion, and
coded residuals
– At the sample level, quantizer-dependent thresholds can turn off filtering for every
individual sample
– At the tiling level, a specially strong filter for macroblocks with very flat
characteristics almost removes „tiling artifacts“
40
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 – Principle of Deblocking Filter
∗ One-dimensional visualization of an edge position
– Filtering of p0 and q0 only occurs if
where B(QP) is << than α(QP)
– Filtering of p1 and q1 ioccurs if also
∗ (QP is the MPEG quantization parameter)
11
41
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Order of Filtering
∗ Filtering can be done on macroblock basis: immediate after MB decode
∗ First the vertical edges are filtered, then the horizontal edges
∗ Bottom row and right column of a macroblock are filtered when
decoding the corresponding adjacent macroblocks
42
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC –
Subjective Result Deblocking Filter (Inter)
Without filter with H.264/AVC Deblocking
∗ Highly compressed decoded inter picture
∗ Significantly reduces pred. residuals (0.28bpp)
43
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 –
Performance Comparison – (1) ∗ Test of different standards (IEEE Tr. CSVT, July 2003)
– Using same rate-distortion optimization techniques for all coders
– Streaming test: high latency (B frames included)
– Real-time conversion test: no B frames
– Entertainment-quality application test: SD & HD resolution
– Several video sequences per test
– Comparison of 4 codecs:
• MPEG-2 (Main Profile high-latency/streaming test only)
• H.263 (High-latency profile, Conversational High-Compression profile,
Baseline profile)
• MPEG-4 Visual (Simple Profile and Adv. Simple Profile with & without B pict.
• H.264/AVC (Main Profile and Baseline Profile)
44
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Performance – (3)
Test Results Streaming Application
12
45
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Performance – (4)
Example Results Streaming
46
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC –
Performance -(6)
Subjective
Comparison
47
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
AVC / H.264 – Performance – (7)
Result Entertainment QualityConclusion:
H.264 / AVC
saves about 50%
bit rate
compared to
MPEG-2 ..!
48
Fac. EE SPS-VCA
PdW / 2018 Multimedia Video Coding & A / 5LSE0
Module 11 MPEG-4 AVC – H.264
H.264 / AVC – Conclusions – (1)
∗ Video coding: also hybrid video coding, but many differences!
– similar ‘in spirit’ to other standards but with important differences
∗ New key features are:
– Enhanced motion compensation (multi-references)
– Small blocks for transform coding (4x4, 4x8, etc)
– Improved deblocking filter
– Enhanced entropy coding (CAVLC and CABAC)
∗ Substantial bit-rate savings (up to 50%) and complexity
– relative to most other standards for same perceptual quality
– Encoders 3-4x more complex than MPEG-2, decoders 2-3x complex
– Standard within both ITU-T VCEG and ISO/IEC MPEG!