Slide 1

Integrating media playback on embedded devices. DTG HTML5 workshop, London, 3 rd July 2015

Slide 2

Who am I? alex.ashley@youview.com uk.linkedin.com/in/asrashley github.com/asrashley

Slide 3

Page composition

Slide 4

The user sees

Slide 5

The browser is compositing

Slide 6

The hardware is compositing

Slide 7

Typical QtWebKit Media Pipeline

Slide 8

Our Device Media Pipeline

Slide 9

One decoder === one video element Or does it?

Slide 10

Mapping to HTML media elements Direct mapping between elements and hardware resources is problematic Removing a video element from the DOM doesnt instantly clean up its resources Application might want to preload a video while another is playing

Slide 11

Removing a video element $(vidElt).off(); /* unbind all event listeners */ vidElt.pause(); vidElt.src=''; /* remove the source */ try { delete vidElt.src; } catch(e){ } vidElt.load(); /* force source change */ /* remove from DOM */ $(vidElt).remove();

Slide 12

Back-to-back videos Use cases Pre-roll branding idents Advert insertion Seamless start of next episode Options Removing one video before loading next Changing source URL of video element Media Source Extension Multiple video elements

Slide 13

HbbTV 2.0 multiple video elements The terminal shall support at least one HTML5 media element that is playing together with at least two HTML5 media elements in a paused state The terminal may use hardware audio and video decoders to decode and render and HTML5 media elements.

Slide 14

Multiple video elements, single decoder Main video HTML application Advert h/w decoder h/w

Slide 15

Slide 16

Video resizing Not all platforms support moving / resizing video elements There might be hardware limitations on accuracy Granularity of 4, 8, 16 pixels Probably a good idea to avoid animated moves or resizes Sync between video and graphics planes might not be perfect Might trip over race conditions in underlying implementation

Slide 17

Position reporting Ad insertion needs accurate reporting of playback position To find the correct point to pause main content HTML position reporting needs to match position of video on screen From hardware presentation timestamp Needs to work for live streams Where timecodes dont start at zero, but HTML5 requires that they do

Slide 18

Seek For live streams, need to accurately report seekable range Needs support from HLS/DASH/PDL implementation HTML5 applications can be quite challenging in their use of trick play Overlapping seeks Multiple pause/play transitions during seek Seek to end while paused Difficult to create test harnesses

Slide 19

Other media events stalled Binding to FIFO empty signal from hardware error Network errors Codec errors DRM errors Metadata events loadedmetadata canplay durationchange resize

Slide 20

Slide 21

Recommendations Have only one media element in the playing state There might be only one hardware decoder Avoid using opacity on a video element Graphics under video might not be supported Avoid video inside canvas elements Platform might not allow access to decoded video Clean up after yourself Media elements consume scarce resources Keep it simple when moving/resizing video elements Try to avoid lots of state changes during seek If possible, store up the changes until the seeked event

Slide 22