fabio sonnati max2010 h.264 encoding strategies … encoding strategies for all screens ......

© 2010 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.

H.264 Encoding strategies for all screensFabio Sonnati | Adobe Community Professional

October 27, 2010 – Los Angeles


Introduction

H.264 | Encoding strategies for all screens

� Fabio Sonnati - Freelance media application architect, ACP since 2006FMS developer and beta tester since 2003 with expertise in VideoEncoding optimizations. Develops Video encoding & delivery platforms for many media companies.

Blog: http://sonnati.wordpress.comEmail: [email protected]


Introduction

H.264 | Encoding strategies for all screens

� Maximize “video experience” is a complex and specialized work

� Today it is even more complex because Flash Player is available on Multiple Screens: Desktop, Mobile, Tablet, STB.

� The problem is multi-variable: you have to find a balance betweenencoder parametrization, resolution, bitrate and desired quality toaddress the requirements of every scenario.

� On top of that you have to know some best-practices and tips & tricksto maximize user experience and QoS in Flash Player.


Agenda

� H.264 – The “de facto” standard

� Video encoding standards overview

� Video Compression Principles

� The H.26x family – H.264

� H.264 vs VP8

� Encoding Best Practices

� Understanding H.264’s parameters

� Encoding Strategies

� Pre-filtering

� A Resolution-Bitrate mix for each Screen

� Delivery Best Practices

� Playback recommendations

� Dynamic Streaming

� Tips & Tricks

� Q&A

4

© 2010 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5

Part I – H.264 the “de facto” standard


Video encoding standards - overview

� ITU and ISO are the formal organizations for video codec standardization

� In the last two decades ITU and ISO have standardized the most usedaudio and video encoding standards like MPEG2, MPEG4, H.264

� H.264 has been a joint effort to produce a long term technology in video encoding for a wide range of applications.


Video compression principles

A generic video encoding is performed :

� Dividing each video frame into blocks of pixels (usually 8x8) so that processing of video frame can be conducted at block level.

� Applying psycho visual models to selectively reduce information (high spatial frequencies) less important for human vision.

� Exploiting spatial redundancies that exist within the video frame by predicting each block from surrounding blocks (spatial prediction)

� Exploiting temporal dependencies that exist between macro blocks (usually 16x16) in consecutive frames, so that only changes between frames need to be encoded. This is accomplished by using motion estimation and compensation.

� Using entropy coding theories to reduce the average amount of bits needed to represent information (VLC, variable length coding).


Video compression principles

Types of Video Frame :

� I-frames

� Encoded only spatially (Intra-frame). I-frames (Keyframes) are “self contained” and used for stream accessibility.

� P-frames

� Predicted from previous reference frames using motion estimation and compensation (Inter-frames). They are not self-contained and form a chain of references.

� B-frames

� Motion vectors and other data areBi-directionally Interpolated from previousand next reference frames (Inter).B-frames can achieve the highestcompression but must be used properly.


The H.26x Family – H.264

� New transform design (4x4 and 8x8 blocks)

� Several Intra frame prediction modes

� Multiple reference frames

� Accurate motion compensation

� Efficient B-Frames

� In-loop deblocking filter

� Context-adaptive entropy coding (CABAC)

H.264 contains several new features that allow it to compress video much more effectively than older H.26x standards:



Codec “Profiles”

Profiles define exactly which techniques and strategies can be used by the encoder and the decoder. Simpler profiles require less processing power and

less memory but achieve a worse quality/bitrate ratio.

� Baseline Profile (BP): Primarily for lower-cost applications with limited

computing resources, this profile is used widely in mobile applications(No support for B-Frames, CABAC and i8x8)

� Main Profile (MP): Originally intended as the mainstream consumer profile for

broadcast and storage applications, the importance of this profile faded when

the High profile was developed for those applications.

� High Profile (HiP): The primary profile for broadcast and disc storage

applications, particularly for high-definition television applications (this is the

profile adopted into Blu-ray Disc).



Codec “Levels”

� Levels define the max resolution of frames, max required memory, max local bitrate and buffering. Levels are important for device compatibility. Usually the baseline profile is used with a level up to 3.1, the main profile with a level 4.1 and the high profile with levels up to 5.1

� The good new is that Flash Player supports every levels and every profiles so the best is to use High profile with level 4.1 for HD or 5.1 for Full HD. Be aware that on mobile devices very often only lower profiles and levels are hardware accelerated.


H.264 and VP8

VP8 was originally developed by On2, the authors of VP6.Recently Google has acquired On2 to donate VP8 technology to open

source community (WebM), so VP8 is the most advanced “open” codec

today (replacing Theora aka VP3). WebM aims to become the reference

codec for HTML5 video.

In some areas it is very similar to H.264 but with less options and the lack of

B-Frames. This alone can account for a 20% less efficiency compared to

H.264.

However, differently from H.264, it is completely free so Adobe will probably

support it in a future release of Flash Player.

Note: Recently Google launched also WebP initiative to substitute JPG using the Intra

frame compression of VP8


H.264 for image compression

Details of a 1024x576 picture compressed to 50Kbytes using H.264 and JPG

H.264 can reduce size of 40-50% with the same quality.


Part II – Encoding Best Practices


Encoding Strategy

How to maximize the “video-experience” for all screens ?

Define a flexible Encoding Strategy following these points:

� Encoding best-practices

� You need to master your encoder of choice

� Pre-filter source video

� Choose the best resolution-bitrate mix for each scenario

� Delivery best-practices

� Use dynamic streaming (Strobe - OSMF - Custom)

� Optimize video playback in the player


H.264 – Encoders selection

� Desktop encoders

� Adobe Media Encoder CS5

� MainConcept’s Reference

� Sorenson’s Squeeze

� Apple’s QuickTime

� nVidia Elemental Accelerator for Premiere (CUDA)

� Batch transcoding

� Adobe Media Encoding Server

� Carbon coder

� Flip factory

� Live encoders

� Flash Media Live Encoder

� Inlet – Digital Rapids – View Cast – Tandberg – Atheme …

� The open source alternative: FFmpeg


H.264 – Profiles & Levels inside Flash Player

� The H.264 decoder (software) implemented in Flash Player is verygood. Supports baseline, main, high, high10 profiles and level up to 5.1. In the past you might simply target high profile and 4.1…

� But with the rapid spread of Flash on multiple screens (Desktop, TV, Tablet, Mobile), you need to define a dedicated encoding and delivery strategy for each scenario. For example:

Encoding for Desktops & TV :

High profile, level 4.1

Higher resolutions (up to FullHD)

Higher bitrates (megabits)

Encoding for Mobile :

Baseline profile, level 3.1

Lower resolutions (sub HD)

Lower bitrates (hundreds of Kbps)


Understanding H.264 Parameters

� Resolution & Bitrate:

� (see next section)

� Bitrate allocation:

� CBR for streaming and dynamic streaming

� VBR for progressive downloading

� Multi-pass encoding:

� Use always 2 pass

� Frame rate:

� Use original framerate

� B-Frames:

� 3-5, Dynamic placement, use b-pyramid (Note: baseline does not support B-frames)

� IDR-Interval (Keyframe distance):

� Good accessibility (in progressive download) or in Dynamic Streaming : 60 – 120

� Streaming or low accessibility : max 300

� Enable Dynamic placement except for Dynamic Streaming



� Number of reference frames

� Depends by level and resolution . Suggested 2-5

� Motion estimation and compensation

� Use biggest blocks and lower accuracy for test, improve for final encode

� Rate Distortion Optimization (RDO)

� Enable only for final encode

� Adaptive Quantization

� Use always

� PSY Optimizations

� Use in high resolution when encoding at medium bitrate.

� Adaptive 8x8 tansform

� Supported only by high profile

� Deblocking

� Mandatory

� Entropy Coding

� Enable CABAC (Baseline does not support CABAC)



� Example of parametrization

Adobe Flash Media Encoder Server.

Target: Desktop, slow

encoding



� Example of parametrization

Adobe Media Encoder CS5

Target: Mobile

Format – multiplexing: F4V

Video – resolution (depends):For 16:9 content: 512 × 288

For 4:3 content: 480 × 360

Frames per second (fps):Same as source footage

Profile: Baseline

Level: 3.1 (3 reference frames will be used)

Bit rate: VBR, two-pass

Target bit rate: 300–550 kbit/sec. depending by

video "complexity"

Maximum bit rate: (1.5 - 1.6) × target bit rate

Keyframe (depends):

For progressive downloading: fps × (2 - 4)

For streaming from Flash Media Server (because of

the enhanced seek capability): fps × (6 - 8)

Max quality



� FFMPEG – The Swiss Army Knife of video encoding

� Open source command line transcoder

� Supports several audio - video formats and also H.264 (x264 library)

� Used for server side batch encoding by video platforms (like YouTube)

� Not easy to master but very powerful

� GUIs: HandBrake, MediaCoder, StaxRip …

Example of [email protected] encoding via command line:

FFmpeg.exe -i <in file> -acodec libfaac -ar 48000 -ab 64k -s 640×360 –r 25 –g 100 -vcodec

libx264 -b 400k -flags +loop -cmp +chroma –partitions +parti4x4+partp8x8+partb8x8 -

subq 9 -trellis 2 -refs 1 -coder 0 -me_range 16 -keyint_min 25 -sc_threshold 40 -i_qfactor

0.71 -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -level 30 <output

file>


Pre-filtering

Denoising

Pre-process accurately the source is very important to maximize the encoding efficiency.The aim is to reduce video noise, adjust color levels and de-interlace.

The video noise is very frequently present in video clips (due to low lights, film grain, poor cmossensors). It lowers the efficiency of encoding because the encoder tries to reconstruct the noise as a very fine detail. It is very important to reduce video noise with proper filters. The best are the “3D”filters.

Note: Mobile resolutions are less problematic; resizing to a lower output resolution acts as a spatialdenoising filter (see next slide).


Pre-filtering

Resizing

Resizing is very important especially when encoding at multiple resolutions for Dynamic Streaming.Below an example of different detail retention of three major resizing algorithms.

Bilinear filtering reduces details more than Bicubic. Lanczos offers the sharpest result.

Be aware that reducing resolution too much in a single pass can sometimes lead to aliasing, and aliasing can harm compression efficiency. Bilinear is less subject to this side effect.

Bilinear resize Bicubic resize Lanczos resize


Pre-filtering

Deinterlacing

• SD sources are very often interlaced, and 1080i

sources are interlaced. It is very important to use

the most professional deinterlacing routine (motion

compensated adaptive deinterlace routines) to

preserve frame resolution and details because

most de-int filters (wave, bob) simply cut off a field

or blend two fields producing a sensible loss in

quality, details and vertical resolution. This also

harms compression efficiency.

(AlgoSuite or MainConcept plugins for Premiere do

the job)


Resolution-Bitrate Mix

Choose the best resolution-bitrate mix for your scenario

The two most important parameters in encoding are frame resolutionand bitrate. The importance of a good balance between resolution and bitrate is often underestimated.

Rate distortion curves:

Example of curve that map

quality on bitrate at 1280x720

for an high complexity source

and a generic encoder.




Unfortunately rate distortion curves depend by source complexity and resolution so you have a problem with up to 4 degrees of freedom:

resolution | bitrate | complexity | target quality


Example of curve that map

quality on bitrate at 1280x720

for high, medium and low complexity

sources





Example of curves that map

quality on bitrate at different

resolutions (high complexity clips)





Example of curve areas that map

quality on bitrate at different

resolutions and complexities.





Optimize the resolution-bitrate mix




Varying (usually Baseline)848x480Mobile

Very good1280x720Tablet

Very good (StageVideo *)Up to 1920x1080TV & Desktop

HW AccelerationResolution



0.6-1 Mbit/s848 x 480

0.8-1.5 Mbit/s1024 x 576

0.4-0.7 Mbit/s640 x 360

1-2 Mbit/s1280 x 720

2-4 Mbit/s1920 x 1080 *

BitrateResolution

Aspect ratio 16:9

As a general recommendation choose dimensions multiple of 16 or 8

Scenario: Encode for the TV & DesktopBig Screen, 720p [email protected] content easily decoded by old HW, broadband

Choose the best resolution-bitrate mix for your video

0.2-0.4 Mbit/s320 x 240

0.3-0.6 Mbit/s480 x 360

0.5-0.8 Mbit/s640 x 480

0.6-1 Mbit/s720 x 576 (PAL)

BitrateResolution

Aspect ratio 4:3



0.3-0.4 Mbit/s480 x 272

0.2-0.3 Mbit/s320 x 180

0.4-0.6 Mbit/s512 x 288

0.5-0.8 Mbit/s640 x 360

0.7-1.2 Mbit/s848 x 480 *

BitrateResolution

Aspect ratio 16:9

Scenario: Encoding for MobileSmall Screen, heterogeneous support for HW acceleration, narrowband.

I suggest to use [email protected] for maximum support to HW acceleration.


0.2-0.4 Mbit/s320 x 240

0.3-0.6 Mbit/s480 x 360

0.5-0.8 Mbit/s512 x 384

0.6-1 Mbit/s640 x 480

BitrateResolution

Aspect ratio 4:3

Note: in mobile scenario, at low bitrates it’s possible to halve FPS



Advanced Topic: Define a dynamic encoding strategy

Use a content adaptive encoding strategy to define encoding parameters on the

fly depending by content complexity.


Static strategy: fixed bitrate for every

complexitySimple adaptive: bitrate changes

according to complexity

Complex adaptive: bitrate, resolution,

parameters and filters change


Part III – Delivery Best Practices


Playback recommendations

� What delivery technique ?

Flash Player 10.1 supports several delivery techniques that can match any cost /

security / portability requirements

� Progressive Download

� RTMP Streaming

� RTMP Dynamic Streaming

� HTTP Dynamic Streaming

� Peer Assisted Streaming


Playback recommendations

� Which PLAYER ?

� Completely custom (low level Flash Player API)

� OSMF (Framework)

� Strobe or Flash Media Playback (turn-key solutions)

� Generic recommendations:

� Use a 16:10 aspect ratio for the player

� Don’t use wmode = trasparent

� Detect device and serve dedicated optimizations

and parametrizations for each screen

(i.e. send HD video on a mobile is crazy)

� Use Dynamic Streaming

to maximize QoS


Dynamic Streaming

How to improve user experience and QoS ?

� RTMP & HTTP Dynamic Streaming

FP10.1 can provide a superior user experience switching between different

bitrates depending by the network condition of each end user.

FP10.1 offers a QoS Api which gives detailed information about the stream, the

bandwidth and the video decoding process. With such info it is possible to

choose the best stream to assure the highest QoS in that moment.

The DynamicStream class or better OSMF implements all the heuristic for the bitrate switching process and the best stream is served depending by available

bandwidth, processing power and screen resolution.


Dynamic Streaming

0.6-1 Mbit/s848 x 480

0.8-1.5 Mbit/s1024 x 576

0.4-0.7 Mbit/s640 x 360

1-2 Mbit/s1280 x 720

BitrateResolution

Desktop 16:9

0.3-0.4 Mbit/s480 x 272

0.2-0.3 Mbit/s320 x 180

0.4-0.6 Mbit/s512 x 288

0.5-0.8 Mbit/s640 x 360

BitrateResolution

Mobile 16:9

Dynamic Streaming – encoding recommendations

� Use fixed length GOP (2-4 seconds)

� Use 2-pass CBR (or “light” VBR es: VBV buffer = gop length)

� Use same audio settings for each stream

� Find a balanced set of resolutions - bitrates

1-2 Mbit/s1280 x 720

0.8-1.5 Mbit/s1024 x 576

0.6-1 Mbit/s848 x 480

0.3-0.4 Mbit/s480 x 272

0.4-0.6 Mbit/s512 x 288

0.2-0.3 Mbit/s320 x 180

0.5-0.8 Mbit/s640 x 360

BitrateResolution

Multiscreen 16:9

[email protected]

[email protected]


Tips & Tricks : Post-filtering

� Using high compression ratios progressively reduces details (starting from small features

and film grain) and pictures appear to be flat, edges too much smooth and colors washed-out.

Post-filtering can help to “restore” lost details.

� We can use simple (built-in convolution matrix) or very complex filters (pixel bender based).

to restore lost details: A common “sharpen kernel” can be a good start.

� It’s important to understand that this is only a “perceptual” restore and is very CPU

intensive. Use QoS API (FP10) to adaptively use this enhancement (disable on mobile).

// "video" is your video object istancevar filter = new flash.filters.ConvolutionFilter(); filter.matrixX = 3;filter.matrixY = 3;filter.matrix = [-1, 0, -1, 0, 8, 0, -1, 0, -1];filter.bias = 0;filter.divisor = 4;video.filters=[filter];



// Init

var baseX:Number = 1.3;var baseY:Number = 1.3;var numOctaves:Number = 1;var stitch:Boolean = true;var fractalNoise:Boolean = true;var channelOptions:Number = BitmapDataChannel.BLUE|BitmapDataChannel.RED|

BitmapDataChannel.GREENvar grayScale:Boolean=false;var offsets:Array = new Array(new Point(), new Point());var bitmapArray:Array = new Array();var frameNumber:Number = 12;

for (var i = 0; i<frameNumber;i++) {var bmpData:BitmapData = new BitmapData(1280,720);var bmp:Bitmap = new Bitmap(bmpData);bmp.alpha=0.20;bmp.blendMode="overlay";bmpData.perlinNoise(baseX, baseY, numOctaves, Math.random()*65000,stitch,

fractalNoise, channelOptions, grayScale, offsets);bmp.visible=false;bitmapArray.push(bmp);

}

// Noise video sequencesetInterval(alternate,80);var altcnt=0;function alternate() {altcnt++;if (altcnt>=frameNumber){altcnt=0};for(var i=0;i<frameNumber;i++){bitmapArray[i].visible=false;} bitmapArray[altcnt].visible=true;bitmapArray[altcnt].alpha=(0.14+0.06*Math.random())*1.5;

}

Creation of a lookup table of pre

rendered “grain frames” using

perlin Noise.

Generation of an animated

sequence overlayed on video

during playback

Synthetic Grain

Also Film Grain can be restore

“perceptually” overlaying a

sequence of synthetic noise over

video.



Original 1280x720

5 Mbit/s

Encoded 960x544

250 Kbit/sUpscale to 720p

+ Sharpen

+ Synthetic Grain

Encoding’s degradation Restoration inside Flash Player

More info : goo.gl/czOD


Turn in your Surveys for a chance to WIN!

� Hand in your surveys to the room monitors

� One survey will be selected as a winner of the Adobe Press e-book

Adobe Flash Platform from Start to Finish: Working Collaboratively Using Adobe Creative Suite 5

� Winners will be notified via e-mail at the end of each day

43


Q & A

Fabio [email protected]://sonnati.wordpress.com

fabio sonnati max2010 h.264 encoding strategies … encoding strategies for all screens ......

Documents