system architecture and api documentation...

System Architecture and

API Documentation

Update

D4.4

MPAT File ID: D4.4_v1.1.docx

Version: 1.1

Deliverable number:

D4.4

Author: Jean-Claude Dufourd (ParisTech)

Contributors: Benedikt Vogel (IRT), Miggi Zwicklbauer (FOKUS) , Stefano Miccoli (FINCONS), Louay Bassbouss (FOKUS)

Internal reviewers:

Fabian Schiller (IRT)

Ma

Work Package: WP4

Task: T4.2

Nature: R – Report

Dissemination: PU – Public

Status: Living

Delivery date: 30.06.2017

D4.4 System Architecture Update

1

Table of Contents

Introduction 3

Terminology 4

Application Model 4

Application 4

Component 4

Page 4

Layout 5

State 5

Style 5

Dynamic Server 5

Front End 5

Back End 5

Project Architecture 6

Editing Phase 7

WordPress Interface 9

MPAT Plugin and Theme 9

MPAT Post Types 9

MPAT Data in the WP CMS 9

Application data 9

Page data 9

Component data 10

Layout data 11

Timeline data 11

MPAT Internal Services 12

Page REST API 12

Layout REST API 12

Options REST API 13

Software Architecture 13

Front End Architecture 13

Introduction 13

MPAT Application 13

Website Application 14

Slideflow Application 14

Timeline Application 15

Preview Area 16

Progress Bar 16

Editable Timeline 16

Property Sheet 16


2

Buttons 17

About Time 17

HbbTV Library 18

Second Screen API 18

Key Management 18

Stream Event Manager 19

Media Time Manager 19

Back End Architecture 19

User Interface 19

Navigation Schemes 19

Second Screen API 20

Import API 20

Asset API 20

User roles and permissions 20

Dynamic Server Architecture 24

Second Screen Support 24

Identification of device/user 24

360° Video support 25

Ad Insertion/Replacement 26

Ad insertion 26

Ad replacement 27

Asset Converter 28

Edge Server 30

Deployment/Playout/Testing 30

Glossary 31


3

Introduction

This document describes the architecture of MPAT. It starts by outlining the terminology, then describes the content architecture (a.k.a. Front End). Following this, an overall architecture is described, followed by the WordPress architecture (a.k.a. Back End). Finally, the Dynamic Server architecture and the Edge Server architecture are similarly described. Together, these elements form the MPAT architecture.

This document is an update of the initial D4.2 document after one more year of work.


4

Terminology

The generic difference between a model and an instance is the same as between a class and an object in object oriented programming.

Application Model

Created by App Creator.

An MPAT Application Model is a template for an Application, in a broader sense than just a generic application with placeholders. An Application Model contains a set of allowed Page Models. An Application Model has a Style which is the default Style of its Page Models.

An Application Model can be of one of three types:

● Website: Application Models of that type are used to create sets of Pages that use hyperlink-based navigation. There is no timing and no management of stream events in these application models.

● Pageflow: Application Models of that type are used to create sequential sets of Pages that are usually navigated interactively, as extended slide shows.

● Timeline: Application Models of that type are used to create sets of Pages that are triggered by a combination of stream events (broadcast time), interactivity, wall clock time and media time. This is the most general and complex type of application models.

The type of an Application Model is the first thing that is chosen and it cannot be changed. Available features in the editing flow are determined by that choice.

Application

Created by the Content Creator from an Application Model.

An MPAT Application is a coherent set of Pages implementing the interactive part of a TV show. An Application is specific to a TV show, complete with relevant text and media, and ready to be deployed. The Application may have multiple Styles and multiple Layouts.

Note: the interactive part of a TV show may be constructed with multiple Applications, and the viewers may not see the difference if these Applications are visually coherent.

Component

Manipulated by Content Creator.

Potential Components include a menu, a dialog, a vote, a popup, etc. These are the building blocks; the elements that are manipulated by the Content Creator. A component is an instance of a Component Model.

Page

Created by Content Creator from a Page Model.

A Page represents a single (sub)-page of an (MPAT) Application.


5

Layout

A Layout is a policy for spatial organization of components in a Page or Page Model. A Layout has a set of boxes that can receive components.

State

A State is a frontend representation of a layout box. The same box can specify multiple states to allow higher interactivity within the page. Each State contains a component and multiple states for the same box can specify different type of components.

Style

A Style is a set of graphics and text attributes and choice of icons (visual appearance).

Dynamic Server

Node.js server that will answer the dynamic requests from interactive TV content served using MPAT, handle interaction with the companion screen framework and offer synchronization functionality.

Front End

The Front End is the part of the WP-related MPAT code that is dedicated to the production of the actual MPAT Application, HTML, JS, CSS, images and other resources.

Back End

The Back End is the part of the WP-related MPAT code that is dedicated to the management of the authoring UI, link with the WP database, etc.


6

Project Architecture

MPAT will reuse WordPress functionality for authoring, storage and production of interactive TV applications, including those built for the HbbTV platform. MPAT extends WordPress both from the inside (extensions and plugins) and from the outside, by providing complementary new tools and interfacing with existing tools.

As shown in the Figure 1, the project architecture has 3 primary internal components:

Figure 1: Project Architecture

The first component is the set of plugins and extensions of WordPress to provide:

● Production of interactive TV applications, including HbbTV ● Authoring user interface ● Serialization of MPAT data into the WP database

The second component is a dynamic server. This server provides:

● Support for additional MPAT features, such as user management or integration with social networks

● Support for application specific extensions, such as dynamic content

The third component is external to the project: the content server. Broadcasters have existing content servers, hosted either internally or externally. These can also be augmented with external CDN support. It is usually enough to have a URL pointing to the content to interface with this third component. When this content server is organized around “assets”, there is an additional layer of interfacing necessary. Assets are composite objects including various related media together with associated metadata. One example of asset metadata is TV Anytime. MPAT is going to provide interfaces from TV Anytime asset management to the authoring process.


7

Editing Phase

There are two distinct phases when creating an MPAT Application. The first of these is the Editing Phase. This is where the application is created, and involves the participation of a number of users.

Figure 2 Editing Phase

In Figure 2, within the WordPress box, “MPAT JS Components” (dark pink) represents the Front End, i.e. the part of the WordPress code that will produce the HTML, CSS and JS of the MPAT applications.

MPAT PHP Components (green) includes all the modules responsible for everything but the Front End:

● Authoring User Interface: specified in WP2 and WP3, ● Interface with the WordPress DB: to serialize all the MPAT specific information ● Event API: to help the author with setting up the stream events part of the debugging ● Second Screen API: to manage the authoring part of relationship between the TV

application and the second screen application ● Import API: to help the author import TV Anytime Assets from the broadcasters’ asset

managers, for use during authoring and debugging ● Asset API: to help the author with managing assets that are not in sophisticated asset

managers

In the Dynamic Server, nine modules are identified:

● Second Screen Solution (used in runtime) ● Asset Manager

o Video transcoding ● 360 video support (does not appear in figure) ● Ad Insertion (does not appear in figure) ● Communication, as support for other features (does not appear in figure)


8

The remaining elements are:

● The Test Playout: the test playout is a system that allows the realistic (but not final) playout of interactive TV content, such as HbbTV; one example of such a system is a combination of multiplexer software with a DVB modulator.

● The Test TV: it is an interactive television device, compatible with HbbTV for example. This may be either a TV or a set top box with a screen.


9

WordPress Interface

This section describes the “interface” between MPAT and WordPress. MPAT reuses features of WordPress and this section describes which features and how, for example, what is the format of the data saved by MPAT and where in the WordPress CMS is it stored.

MPAT Plugin and Theme

At this time, there are many hard-coded links and dependencies between the MPAT Plugin and the MPAT Theme. The two are a set and cannot be used independently.

MPAT Post Types

An MPAT page is a WP post of type page.

An MPAT layout is a WP post of custom post type page_layout.

MPAT Data in the WP CMS

Application data

Application data consists in information stored in the WP options table, under the key mpat_application_manager:

● navigation_model: one of website, timeline and slideflow, extensible ● root_application_url: the entry point of the application ● slideflow_orientation: vertical or horizontal ● slideflow_settings: this describes which pages of the current WP site are used and in

which order ● language: the language of the MPAT application, a 2 character code.

Page data

● Page data includes many things. It is stored in the WP postmeta table with key mpat_content and the post id of the page. Page data in the WP post standard properties are:

● post_id is reused as the page id ● post_title is reused as the page name ● various WP post dates, states and status can also be used

Specific MPAT page data consists in:

● background: string, defining a background color or image ● componentStyles: an array of styles for the boxes in content, including:

○ a generated string starting with _ for the component name ○ an object with one or more of color, padding, border, border radius, text color;

the values use typical CSS syntax. ● content: an array of boxes, with keys starting with _, and describing a component

with an array of component states, each state with: ■ stateTitle: string ■ data: component specific data ■ navigable: whether the component is navigable ■ dontVisualizeFocus: boolean ■ scrollable: boolean ■ type: string, one of text, scrolledtext, image, video, menu, launcher…

● layoutId: a number, the post id of the layout used by the page ● defaultActive: the component which is active at the start of the page


10

Component data

Component data within page data is specific to each component type:

● gallery: image gallery ○ arrows: boolean ○ dots: boolean ○ images: array of elements with:

■ attachmentUrl: URL of the image ■ attachmentId ■ id

● audio: url, autostart and loop ● broadcast: the video broadcast component ● clone: a link to a component in another page; pageId, componentId ● image: imageUrl, postid ● launcher:

○ orientation: one of horizontal or vertical ○ format: one of portrait, landscape, square, squareWithInfo ○ style: one of standard or arte ○ scrollStyle: one of pagination or carousel ○ showPaginationInfo: boolean ○ nCols: number ○ nRows: number ○ listArray: array of launcher elements as:

■ imageUrl: thumbnail ■ title: string ■ role: role of the launcher element ■ appUrl: link ■ pages: link to page ■ description: string ■ icon: content icon, URL

● link: ○ thumbnail: link to the image ○ url: link to follow if triggered ○ cover: boolean

● list: ○ listArray: array of elements as:

■ title: string ■ appUrl: URL to follow if selected ■ description: string

● menu: ○ orientation: one of horizontal or vertical ○ loop: boolean ○ sidemenu: boolean ○ showButtons: boolean ○ listArray: array of menu elements as:

■ appUrl: link to a page ■ description: string ■ remoteKey: key triggering this menu element ■ role: one of link (extensible)

● redbutton: ○ img: URL of the image ○ time: time in seconds until appearance ○ duration: time in seconds until disappearance ○ url: URL of the application to be triggered

● scrolledtext: ○ text: the HTML text


11

○ arrowsPosition: one from ONTEXTRIGHT, ONTEXTLEFT, OUTSIDERIGHT, OUTSIDELEFT, NOARROWS, ABOVEBELOWRIGHT, ABOVEBELOWLEFT

● text: ○ text: the HTML text

● toggletracking: button, enabledText, disabledText ● video: thumbnail, url, autostart and loop

Layout data

Layout data consists in a list of box descriptions for components. It is stored in the WP postmeta table with key mpat_content and the post id of the layout. Layout data in the WP post standard properties are:

● post_id is reused as the layout id ● post_title is reused as the layout name ● various WP post dates, states and status can also be used

Specific MPAT layout data consists in:

● layout: an array of boxes, each with: ○ w: width (in units of 10 pixels) ○ h: height (in units of 10 pixels) ○ x: horizontal coordinate (in units of 10 pixels) ○ y: vertical coordinate (in units of 10 pixels) ○ i: an auto-generated string for the name of the box ○ minW: minimum width (in units of 10 pixels) ○ minH: minimum height (in units of 10 pixels) ○ static: whether the box is editable

Timeline data

Timeline data consists in information needed to produce a timeline application, described futher below. It is stored in two keys of the WP options table: timeline_scenario and dsmcc.

The first contains:

● frontComponent: defines a page that is displayed in front of the timeline. Usage includes a general menu. If defined, contains an object with properties below

○ name: a string ○ guid: the URL of the component ○ blogid: the number of the site this page belongs to (may be different from

current blog) ○ postid: the post id of the page

● backComponent: defines a page that is displayed behind the timeline. Usage includes a page with a streamed video, whose progress events drive the timeline. If defined, contains an object with properties below

○ name: a string ○ guid: the URL of the component ○ blogid: the number of the site this page belongs to (may be different from

current blog) ○ postid: the post id of the page

● ranges: the list of existing ranges, in the order of definition (never changes). The structure of a range is:

○ id: number ○ title: string ○ blogid: a timeline can refer to pages in other sites, hence the blogid (number) ○ postid: number ○ type: one of TimeEvent, MediaEvent, StreamEvent, KeyEvent ○ data: when a StreamEvent, this string defines the stream event payload that

will trigger this range ○ keyCode: when a KeyEvent, this defines which key is expected to trigger this


12

range ○ start: in pixels ○ width: in pixels ○ begin: start expressed in seconds ○ duration: width expressed in seconds ○ url: the URL of the page ○ key: internal

● sortedRanges: the list of indices of ranges, in the editing order (changes with editing) ● marker: the position of the playing time marker

○ pixel ○ seconds

● timeLineLength: width of the timeline in pixels ● timeLineDuration: duration of the timeline in seconds ● fps: number of frames per seconds ● selected: index of the selected range ● plus other internals

The second contains:

● tag: the DSMCC object tag containing the stream event ● id: the stream event id ● name: the stream event name

MPAT Internal Services

There are multiple REST services allowing to create, read, update and delete pages, layouts or options.

To read all pages or layouts or options, execute an HTTP GET on the URL of the service. An array of pages or layouts is returned as a JSON object.

To read one page or layout or option, execute an HTTP GET on the URL of the service with ‘/id’ appended, with id being the numerical id of the page or layout or the name of the option. A description of one page or layout is returned as a JSON object.

To create a new page or layout, execute a POST on the URL of the service (no id), passing the new description as a JSON object in the same format as upon a read. This does not work on options, use PUT.

To update an existing page or layout or option, execute a PUT on the URL of the service with ‘/id’ appended, passing the modified description as a JSON object in the same format as upon a read. Unmodified properties can be omitted.

To delete an existing page or layout or option, execute a DELETE on the URL of the service with ‘/id’ appended.

The REST services require authentication. To use them from outside MPAT, please make use of an authentication plugin: JSON Basic Authentication is an example useful for development.

Page REST API

The base URL of the page REST API is <blogurl>/wp-json/wp/v2/pages. This service is the standard WP REST service, with an additional configuration to include the mpat_content key of the postmeta.

In JavaScript, the class offering page REST IO is PageIO.

Layout REST API

The base URL of the layout REST API is <blogurl>/wp-json/mpat/v1/layout. This service is a specific MPAT-developed service, modeled after the page service.

In JavaScript, the class offering layout REST IO is LayoutIO.


13

Options REST API

The base URL of the option REST API is <blogurl>/wp-json/mpat/v1/option. This service is a specific MPAT-developed service, modeled after the layout service.

In JavaScript, the class offering options REST IO is OptionIO.

Software Architecture

The underlying WordPress software architecture is the foundation of the MPAT software architecture. A MySQL database provides the storage. A content management system (CMS) layer is implemented in PHP on top of the database. All of the server side WordPress code is in PHP, in a complex blog layer with dozens of hooks, to modify the behaviour.

The lower layer of MPAT is in PHP:

- extracting data from the CMS for authoring - building the editor - dynamically reacting to editing actions, in particular feeding data back into the CMS - extracting data from the CMS for publishing

The first three are part of the back end.

The last constitutes the front end.

Both frontend and backend work on the data model described in the MPAT data section. Both ends rely on React and redux to create, on the client side, complete pages from the model data.

Front End Architecture

Introduction

The Front End is the part of the WP-related MPAT code that is dedicated to the production of the actual MPAT Application, HTML, JS, CSS, images and other resources.

There is a distinction between what the viewer perceives and what the designers and developers do. So the initial perception of one application for one TV show is relevant for the viewer, but not for the designers and developers in MPAT. The viewers may see a visually coherent set of pages, but MPAT may choose to implement this with multiple “applications”. MPAT Applications have an internal coherence determined by the capabilities of the system, and multiple MPAT applications may be associated to create the feeling of one “application” for the viewers of a TV show.

The typical structure from a viewer perspective is:

● Hint towards the presence of an application: red button visual, appears and disappears periodically

● Click on red button starts a launcher with various possibilities, including something for the current show (one or more), which is (are) an MPAT application

● The MPAT application for the current show

MPAT Application

An (MPAT) Application is a set of Pages to be distributed with a TV show. We assume on this object:

● A single style ● A single entry point (the URL specified in the AIT) ● A single origin (to avoid CORS issues) or a set of predefined origins (e.g. to include the


14

dynamic server) with suitable CORS headers ● Created from a single Application Model ● Many of the above are managed by the fact that an application resides within a single

WordPress site.

From a React perspective, all applications have (at least) one page shown on the screen. All navigation models defined in MPAT core support async load of contents for smoother UX. Slideflow and timeline applications also have a dictionary of possible pages stored in a JS object: when a link is triggered, instead of fetching the linked page from the Internet, the application looks into its dictionary for the information needed to render the linked page with React. Only new media are fetched. The form of the data in the dictionary follows the same model as the one described in the MPAT data section.

Website Application

The website application has always exactly one page on screen at any time. Navigation between the pages of the application is done by hyperlinks, triggered either by navigation or by clicking on designated keys on the remote control.

Zones in the current page that can be navigated are shown with a white rectangle around them when focused. Cursor keys can change the selected zone by direction: clicking on the left arrow passes the focus to the nearest zone to the left of the current focussed zone. If there is none, nothing happens. Enter is used to trigger the action of the current focussed zone: the action may be to get into another layer of navigation (inside the focussed component).

The web site navigation scheme is based on authoring individual single-screen WP pages and explicitly providing links to navigate between those pages. The author of an application designs individual pages. In most applications, these perform a single function (such as information display, video gallery), but it is also possible to combine multiple components (for example an image and an additional social media message display) within a page. In the predecessor to MPAT (the HAT editor), the possible distribution of components on the screen was limited. Layouts were full-screen, split in two regions (left and right) or three regions (left, middle and right), with one component in each region.

In MPAT, the layout handling implementation is based on React.js to allow an enhanced underlying capability. This provides more flexibility in the placement and sizing of the component, enabling the free positioning of arbitrarily sized rectangular regions on the screen. While there still are a number of constraints (such as the fact that regions are not allowed to overlap), it adds considerable freedom to the placing of components.

Navigation flow between individual pages is arbitrary and needs to be provided by the application editor. This is usually achieved either by making direct key shortcuts to the other pages or by providing selection of other pages from a pull-down menu. The former solution (key shortcuts) is preferred if pages have significant differences, while menu selection is preferable if the pages are conceptually on the same level (for example a list of actor biographies).

Slideflow Application

The slideflow application also has always exactly one page on screen. The slideflow application is actually a restricted version of a website application: all pages are considered “slides”. Navigation with the cursor (left-right or up-down) is dedicated to switching slides. Other keys can be used for interacting with the current page/slide.

Slideflow navigation is based on the concept of spatial navigation between individual screens. Creation of individual screens (WP pages) and their content is identical to the Web Site navigation scheme.

The Slideflow navigation scheme differs from the web Site navigation scheme in that pages are assumed to be either ‘above’ or ‘besides’ each other, so that an implicit navigation concepts, based on cursor keys, exists to navigate to the page ‘above’, ‘below’, ‘left’ or ‘right’ of the current one.


15

The Slideflow navigation scheme is best suited for presentations that have an implicit ‘narrative’ about a topic. Users will in almost every case, unless shortcuts are intentionally provided, need to follow the existing page order, making Slideflow an ideal choice for applications where information is best presented in a specific order, but less suitable where users want to access specific items quickly.

While the architecture doesn’t enforce any preferences in order, it is encouraged to have a narrative flow that goes ‘down’, with pages containing additional details to the right.

The interface automatically presents on-screen arrows to indicate the availability of additional pages that the user can navigate to.

To make the spatial arrangement of the pages clearer to the user and to enforce the visual metaphor, it is desirable to scroll (flow) between individual pages when moving between them, as opposed to immediately switching between them. Technically, smooth scrolling is difficult to achieve on earlier or low powered HbbTV devices. For such devices other indicators will be used to visualize the spatial relation and flow between the pages.

Timeline Application

The timeline application may have 0, 1, 2 or 3 pages shown together at any time. Behind all the pages is the broadcast video, if present. The back page is here to receive a page with a streaming video, for applications where the progress into a streaming video is used to drive another application: e.g. in an educational situation, at some point in the video, a quiz application can be shown on a corner of the streaming video. The main page is here to receive most of the content shown in the timeline, including the quiz application in the previous example. A front page is here to receive a general popup menu, which allows the user to navigate by interaction into the applications shown in the main page.

The Timeline navigation scheme covers applications that are time oriented, depending either on media time in the case of applications such as VoD and catch-up, or on stream events in the case of live broadcasts where the precise synchronization of HbbTV elements is provided through stream events.

The principle of these applications is that they have a duration: immediately after choosing the Timeline navigation scheme, the author should indicate the total duration of the show. This duration decides the time span in which elements can be added and removed, and thus it is difficult to reduce it after elements have started being added.

Another option is the background image to use in the preview area, something that should be representative of the actual show. MPAT elements are designed (as WP pages) and edited in the same UI as the Web Site. MPAT elements cannot be edited directly in the Timeline Editor below.

The Timeline editor looks like this:


16

Figure 3 Time Line Editor UI

Preview Area

The preview area displays:

● The background image ● The MPAT element that should be present at the time shown by the marker in the

progress bar, if any

There is no interactivity, or animation.

Progress Bar

The progress bar shows the duration of the application, and a marker. The marker can be moved by drag and drop along the progress bar. As soon as the marker is dropped, the preview area is updated.

The progress bar also holds the ranges currently defined.

Editable Timeline

The editable timeline shows the time ranges (in various colors) when MPAT elements are shown.

Possible interactions are:

- Click on a time range to select it: the time range is highlighted, its attributes are shown in the property sheet

- Click on the selected time range or in an empty zone: deselects the time range, empties the property sheet

- Drag the body of a time range: changes the starting time - Drag the last 10% of a time range: changes the duration of the time range

Property Sheet

The property sheet (titled event editing) shows the characteristics of the selected time range and the associated MAP element:

● Starting time or stream event


17

● Ending time or stream event ● MPAT element

Possible interactions are:

● Change one of the times with another value ● Change one of the stream events attributes ● Switch from time to stream event or back

Buttons

In the buttons zone (pink), there are four elements:

- A MPAT element selector: this selector holds the list of all MPAT elements already defined; the author chooses which element to add in the selector, then presses the New button below.

- A “New MPAT Element” button: creates a new time range in the timeline, starting in the first empty time and with maximum possible duration, i.e. no overlap with the next time range.

- A “Remove MPAT Element” button: can be used to remove the selected time range, inhibited if no time range is selected.

- A Publish button: pressing this button generates the page and playout information, including stream events description if relevant.

About Time

The consideration of time is necessary in a subset of applications. When time related editing is present, there may be constraints on the adopted structure for an Application. If elements need to appear and disappear according to the current media time, then one single HTML per Application would be the only possible structure. Components can be in sub-pages (HTML files loaded via iframe, or HTML fragments loaded by AJAX), but one main HTML is loaded and stays “on top” until the end of the use of the Application.

Having one single HTML that is loaded over the whole life-cycle of the Application means that we can keep track of what the user has done, without requiring the storage of such information (using cookies or another alternative solution).

The notion of scene time does not exist in HTML. For example, there is no simple way to trigger the action of some HTML element at a specific time within a show (without StreamEvent support). It is however possible to trigger a HTML element at a certain UTC time, which may be different across devices. The only synchronization which is compatible across a wide range of devices uses the HbbTV standard’s StreamEvents. Existing timing attributes in the HbbTV standard include:

● The time of tune-in or of the beginning of the show, i.e. the first time a script is run on the TV and the date/time logged; this will be different for each TV

● A UTC time; the actual precision is unknown in this context

● The reception of an HbbTV StreamEvent; synchronization between sets can be determined and, if compatible, somewhat guaranteed

● The time of a user interaction

● Any of the above, but including a specific offset

Each new element is a sub-page. This will be served by a web server if constant, or by the backend server if not. It will be included in a target div element, whose content is loaded by an AJAX request. These elements are added and removed by a script reacting to a stream event or key event.


18

HbbTV Library

The (very small) HbbTV Library contains a number of useful components for interacting specifically with HbbTV-compatible devices, including:

● Remote debugger ● Video player handling ● Event handling

○ Key events ○ Stream events

Second Screen API

The Second Screen API allows a companion application on a smartphone or tablet to connect to a HbbTV Application, exchange data and synchronize events. HbbTV 2.0 Terminals support the Second Screen API nativel. As a result, there is no need to include any JavaScript Library in the HbbTV App. The following Companion Screen features are supported in the HbbTV 2.0 specification:

● Launching a HbbTV Application from a companion screen: this uses DIAL (Discovery and Launch Protocol) to discover HbbTV 2.0 Terminals in a local network, and then launch the HbbTV App

● Launch a Companion Screen App from a HbbTV App: the HbbTV App uses a JavaScript API (part of the HbbTV 2.0 Spec) to discover Companion devices and launch second screen App on them

● App2App Communication: The HbbTV 2.0 Terminal provides a WebSocket Server for App2App communication. Each of the HbbTV and Companion Screen Applications establishes a WebSocket Connection to the WS Server on the same channel. The Server forwards the messages between both applications

As all terminals currently available on the market support HbbTV 1.x, we provide a Second Screen Framework called Famium on top of HbbTV 1.x that supports the same features mentioned above with some limitations:

● Both HbbTV and Companion Screen Apps need to embed a JavaScript Lib in order to support the Second Screen API

● The App2App communication Server is hosted in the Cloud. Local App2App communication is not possible

● Local Discovery and Launch is not possible. Therefore a pairing step is required. During pairing, the HbbTV and Companion Screen Apps exchange a token that is stored locally in both Apps. There are different ways to exchange the token e.g. using QR-code, PIN-code or by entering a short URL in the Browser of the Companion Screen

● The Famium API is compatible with the W3C Presentation API

Key Management

Key interaction can be either chosen by the component creator or by the content creator.

Any Component or Navigation Model can register events. To do this, the key must have a priority level and a function to be executed. The priority level has gone through, when a key event is triggered. Also it has to be checked if the key event is registered. All events of a level that match the key are executed and the event is then discarded. If no event is present on the priority level, the next lower priority level is reached.

The component creator also needs to decide which key should reach which function. But he can give also this decision to the content creator.

There are five priority levels for the key events:


19

always

● independent from the app status ● should react every time (like Red Button)

active

● event should be fired when the component is active ● e.g. activate a component by clicking OK (when the smooth navigation is not active)

focused

● function should be done if the component is focused ● e.g. popup of a Hotspot

controller

● function of the Navigation Model ● e.g. UP and DOWN for the Slideflow

global

● independent from the app status ● when no other Component listen to the key ● e.g. Player Controls of the Video Component (play, pause)

Stream Event Manager

Within the Timeline application, the Stream Event Manager listens for Stream Events and calls the appropriate callback for each event received.

The Stream Event Manager has an associative list of stream events to callback. When an

event of matching name is received, the callback is called with the data of the stream event.

Media Time Manager

The Media Time Manager is only useful when there is media playing in the application, and that media is not the broadcast video. The broadcast video does not have a meaningful “currentTime”. Other playing media, such as downloaded/streamed video/audio streams, have a meaningful currentTime which can be used to trigger various scene actions: showing a component, removing a component, etc.

The media time manager operates in a similar way to the stream event manager, but instead of listening to stream events coming from DVB/DSMCC, it listens to the progress of a media and triggers actions upon time cues provided in the timeline.

Back End Architecture

User Interface

The back end user interface is based on React.js. In order to accelerate the production of a working tool for the initial pilot phase of the project, input WP2 and WP3 will be partially considered. This will then be rectified in later revisions of the implementation. In the meantime, we are going to produce a temporary user interface, which incorporates some, but not all, of the desired features.

Different navigation schemes will use different user interfaces. These are described in the following section.

Navigation Schemes

A navigation scheme is one of the first choices made by an author. The choice is made in the general MPAT settings for the whole WP “site”, given that a WP site corresponds to an MPAT


20

application.

There are three currently defined navigation schemes, but there is a mechanism for defining a new navigation scheme in a WP plugin and registering it in the MPAT authoring tool.

In the WP settings, a MPAT settings page is defined. The MPAT settings page has one option, which is the navigation scheme. A function can be called to register a navigation scheme, defined below. After selection of one navigation scheme, the specific options of that navigation scheme are added to the settings page.

Second Screen API

This module generates the MPAT internal description to feed the Second Screen module of the dynamic server with relevant information.

Import API

This module defines interfaces to integrate external asset management systems in MPAT. All standard CRUD operations, namely create, read/retrieve, update/modify and delete/destroy, are supported, enabling editors to manage media assets through MPAT administration UI.

Import functionality is implemented only logically: files continue to reside on external CMS which is in charge of the delivery to final users while metadata are stored in MPAT reflecting TV anytime specifications as instances of video class defined in Asset API module. The module provides dedicated UI to allow content editors to overwrite that information as needed. Also, the Import API module handles synchronization and conflict management between local and remote revisions of each asset.

This module imports TV Anytime-formatted information about a media asset.

● Items: on-demand video, broadcast and even live-broadcast ● Sources:

● Link URL or pre-produced content from CMS of broadcaster ● Dummy for live-broadcast applications (e.g. in a soccer game the overlay’s

position of the goal count is predictable/ the live-broadcast is not, the editor might want to trigger certain actions which are already pre-configured, i.e. player’s stats are shown after a goal was shot)

Asset API

The purpose of this module is to define asset structure and definition in MPAT cms, since WordPress implementation of media content is too simplistic, and to interact with asset converter to manipulate such contents. It is implemented as a WordPress plugin and its main features are:

● Extend WordPress attachment post type to define fields required by MPAT assets, like video metadata and reference to files. Metadata fields definition derives from TV-Anytime specifications. It relies on WordPress API for fields definition, their persistence and representation in administration UI.

● Feed asset converter with assets uploaded from MPAT administration UI to ensure highest coverage of devices with advanced features like adaptive streaming protocols.

● Provide admin UI to control asset converter tasks such as management of scheduled jobs, reprocessing of existing video files and generation of preview images.

User roles and permissions

Assumptions

- Each user can have multiple roles - Standard user roles, with related permissions, are provided by MPAT, while custom


21

roles can be defined with permissions from different standard roles - MPAT provides backend UI to assign roles to users and assign and revoke

permissions to roles

User roles defined in D2.1

- Concept Developer: Concept development & application design, external to MPAT; no user role needed

- API Developer: Feature implementation based on concept; no admin UI operations, no user role needed

- Layout Designer: Definition of the application’s visual look & feel (e.g., styles, templates); user role needed

- Application Creator: Application creation using MPAT authoring tools; user role required

- Content Editor: Integration of journalistic content into the application; user role required - Approval Editor: Approval of application for release/publishing; user role required - Technical Publisher: Roll-out & publishing of the application; user role required - Technical Administrator/Superuser: Manage roles grants and MPAT configuration,

have access to all areas; user role required

Implementation

- WordPress user API will be used (read more) - Ready made plugins to provide administration roles interface will be evaluated (more

likely members plugin) - Custom post statuses required by MPAT app and content creation workflows:

- Pending technical approval: pages created by App Creator which need validation

- Layout ready: pages designed by App Creator and which can be edited by Content Editor

- Pending content approval: pages completed by Content Editor which need validation by Approval Editor

Layout designer

Application Creator

Content Editor

Approval Editor

Technical Publisher

Technical Administrator

View styles X X

Create styles X X

Edit styles X X

Delete styles X X

View templates

X X

Create templates

X X

Edit templates X X

Delete templates

X X

View applications

X X[1] X

https://codex.wordpress.org/Roles_and_Capabilities

https://wordpress.org/plugins/members/


22

Create application

X X

Edit application

X X

Delete own applications

X X

Delete others applications

X

View pages X X[2] X

Create pages X X

Clone pages X X

Access to pages edit form

X X X

Select style X X

Edit style[3] X X

Select template

X X

Add modules X X

Remove modules

X X

Edit module settings

X X

Edit module content

X X

Submit page for “technical

approval” [4]

X X

Approve page structure[5]

X

Submit page for “content

approval” [6]

X X

Approve page content[7]

X X

Publish

page[8] X[9] X X

Access to Asset converter dashboard

X X

Trigger deploy

scripts[10] X X


23

[1] Content editor should see only applications for which he/she was previously authorized

[2] Content editor should see only pages being part of applications for which he/she was

previously authorized, and only with “layout ready”, “draft”, “pending content approval” and “publish” status, NOT “pending technical approval” status

[3] Only few style properties can be changed on page basis

[4] From “draft” to “pending technical approval” status

[5] From “pending technical approval” to “layout ready” status

[6] From “draft” or “layout ready” to “pending content approval” status

[7] From “pending content approval”, to “publish” status

[8] From any to “publish” status (sometimes “approve page content” and “publish” actions

overlap)

[9] Content editor can publish page only when editing previously created and approved, alias

“published” pages.

[10] TBD


24

Dynamic Server Architecture

This is a Node.js server. It includes the following modules:

- Communication - Second Screen - 360° video support - Ad insertion - Asset converter

Whereas WP modules are strictly concerned with production of HTML/CSS/JS for the presentation part, to be used in the browser (TV or CS), the dynamic server modules do all the hidden, server-side work, e.g. sending fragments for dynamic presentations, sending sync information, managing the point of view, managing maps and POIs...

Second Screen Support

The Second Screen backend components are required for HbbTV 1.x Terminals since a local discovery, launch and communication is only supported in HbbTV 2.0. As fallback for HbbTV 1.x devices, the Second Screen Backend is needed. It consists of the following components:

● Device and Connection Manager: It manages all devices (HbbTV Terminals and Companion Screens) connected to the Second Screen Backend. Devices need first to join a room in order to be able to communicate with each other.

● Device Pairing Manager: to connect a Companion Screen to a HbbTV Application, a pairing of both devices is required. During pairing, a room token will be exchanges between both devices and stored in local storage or cookies in both apps. This allows to not repeat the pairing after each launch of the application. The HbbTV or Companion Screen App can remove the token at any time later. After this, the pairing step needs to be repeated in order to connect the devices again.

● App2App Communication Proxy: This component allows establishing proxied bidirectional communication channel between companion screen and HbbTV applications.

Identification of device/user

We need a way to identify a device, e.g. to identify that it is the same device as some time before.

The two things that a server will know about a device that connects are:

- The visible IP - The user agent string

Two TVs of the same brand and model in the same house/LAN with a single visible IP will not be distinguishable, but that may be rare enough to be an acceptable limitation.

Using cookies might be a viable alternative. The device gets a unique cookie upon its first connection. A second device will get another cookie. Note: If cookies are reset, the device will be (mistakenly) identified as a new device.

For companion screens, the problem is similar, with a bigger chance of ambiguity. However, in these cases, it is easier to ask for actual user identification.


25

360° Video support

The 360° Video support in MPAT based on the Fraunhofer FOKUS Cloud-based 360° Video Playout for HbbTV. This Playout performs the rendering of the individual view on server side so that only the selected video content is streamed to the end device. This reduces the bandwidth needed or, put the other way around, allows for a higher quality video delivered on a given bandwidth. It also diminishes the requirements for the end device to simply play back a usual video stream in order to provide the full 360° video experience.

Figure 4 How to watch 360° Video content on TV via HbbTV

In MPAT we implement a 360° Video Plugin to integrate the Cloud-based solution into the MPAT system. The Fraunhofer FOKUS Cloud-based 360° Video Playout for HbbTV consists as depicted in the figure above of the following components:

● 360° Video Cloud Streaming Server: The 360° Video Cloud Streaming Server renders any kind of 360° content and streams the individual view to playback devices such as TVs and mobile devices. The 360° Video Cloud Streaming Server offers a REST API to control the individual view and playback state for each client.

● 360° Video TV Player: The 360° Video Cloud Streaming solution enables Smart and Hybrid TVs to provide 360° Video experience through usual video playback of the output from the cloud streaming server. Users can control their individual view and playback state via TV remote control.


26

Figure 5 How to predict future camera angle with

FOKUS Cloud-based 360° Video Playout for HbbTV

This player is replaced with the MPAT 360° Video Player that will be also customizable for the editor.

Ad Insertion/Replacement

The following section describes a prototypical broadcast and IP-Video handling of HbbTV related ad services. Due to its technical complexity, only the preconditions for setting up this functionality can be provided during the scope of the project. This is mainly the editable time-line described above in conjunction with broadcast stream-events, which provide the capability to synchronize broadcast and VoD.

The gathering of user information and its analysis is outside of the scope of MPAT. As there are already accepted ad-systems, which are specialized in user tracking, the necessary data is handled by external ad service (e.g. Google ads). The ad related functionality implemented in MPAT shall support two use cases:

Ad insertion

This only relates to IP-videos in the HbbTV domain. VoD is to be paused during playback without user-interaction and inserted with targeted IP-video-ad(s). After the insertion, the video resumes to play from the previous paused position. This feature also includes a timer function to set an interval of how often the ad clips are interrupted in respect to the video playback.


27

Figure 6 Ad Insertion

Required functionality:

● External communication with ad server ● Interrupting on-demand-videos and inserting ad-videos ● Time-synched pre-buffering of ad-videos ● Insert options

● Insert at start of VoD ● Setting a time interval for insertion

Ad replacement

The broadcast video clips from an ad-break are overlaid (replaced) with IP-Videos in order to present personalized ads to the viewer. Therefore the ads of broadcast and IP have to be pre-buffered and synchronized in a frame accurate manner. This can be done one by one clip or on the whole ad-break.

Figure 7 Ad Replacement

Required functionality:

● External communication with ad server


28

● Frame accurate synchronization of IP-videos with broadcast using stream events ● Time-synched pre-buffering of ad-videos

Asset Converter

Figure 8 Asset Converter

MPAT allows the content editors to insert videos in applications from several sources, which leads to the necessity of a tool that ensure compatibility with formats and codecs supported by HbbTV and mobile devices, namely the Asset Converter (AC). At the highest level, the asset converter digests video files and converts them with specified video and audio settings to assure compatibility with different interactive TV and mobile devices, considering also latest protocols for adaptive streaming. It relies on several third-party libraries and tools for its operation:

● Composer: Dependency manager for PHP projects ● Monolog: A PHP library which implements PSR3 interface for logging in PHP

applications and allows the asset converter to log messages on multiple streams at the same time.

● Components from Symfony framework: used for tasks like filesystem manipulation, application configuration and generation of a CLI application

● FFMPEG: open source library for video and audio manipulation, used in Asset Converter mostly for video conversion, image extraction and HLS segmentation to achieve adaptive streaming on mobile devices

● MP4Box: A segmenter tool to generate MPEG-dash compliant files to achieve adaptive streaming on HbbTV devices

In the first release, users can interact with Asset converter in two ways, executing commands through the CLI application provided or calling APIs after including the asset converter as a dependency. The integration plugin will use APIs provided by the AC classes to create CMS representations of the converted assets.

Assets can be added to the AC individually, providing their path or URL, or from a given folder. In the latter case, all the compatible files, are imported in the AC. Either way, the AC creates a copy of the original asset in its storage folder to ensure consistency throughout the time and prevent cases where further conversions fail because of missing source file. AC behavior on duplicated assets and folder cleaning can be configured at the application level. Scenarios like automatic digest of assets from third-parties like collaborators and agencies can be easily achieved scheduling a periodic import from folders previously exposed by the FTP server.

The asset converter implements a simple but effective queue system with a lock mechanism while processing the assets. This prevent collisions between encoding jobs and performance degradation caused by uncontrolled proliferation of running processes. The simplest, as well as


29

the most conservative, AC configuration considers one encoding job for one instance of the AC. Due to encoding time and number of requested assets, it might happen that the queue grows faster than AC capability to serve encoded files. In this case, some measures can be adopted. At first, FFMPEG threads number can be tuned accordingly to machine specifications to enable AC to use multiple processors. It is also possible to schedule multiple jobs at shifted intervals, reducing the time between one encoding task and the next, at the cost of more resources used. Another approach could be to setup multiple instances of AC on several machines, sharing database and filesystem across them allows having a single queue management while multiplying encoding capabilities.

While the Asset Converter needs its own database and filesystem, there is nothing to prevent the reuse of those used elsewhere within MPAT. This configuration has some benefits, such as reduced configuration times and also import times, since assets which are already in AC folder are not copied but just added to the AC database. As a downside, mostly if the MPAT CMS is used as a production backend, assets conversion may lead to overloads resulting in a degradation of performance of MPAT applications. In that scenario, it is suggested to setup one or more separated AC instances.


30

Edge Server

An edge server, in a system administration context, is any server that resides on the “edge” between two networks, typically a private network and the Internet. An edge server will be used in the MPAT-Production environment for serving the final HbbTV app in form of HTML/JS/CSS and will act as a Proxy-Server for the communication with the Node.js backend aka dynamic backend.

The Edge Server is a requirement from Broadcasters for security reasons because it will decouple the MPAT-Authoring and the final HbbTV Application. As a side effect the Edge Server will be used to translate requests to the MPAT-Dynamic Server. Typically Services on the Dynamic Server are available on different ports.

Deployment/Playout/Testing

Due to the hybrid approach of the MPAT system, special attention has to be paid to the deployment of the applications. The multi-platform needs to provide data via broadband and broadcast, therefore two separate ways to deliver the data has to be chosen.

The broadcast playout on the other hand needs to specify where the app is located in the web. This is done with the help of the so-called AIT (application information table) included in the MPEG transport stream. There, the start-URL of the designated application is signaled to the TV-client. According to the HbbTV specification the URL is bound to the TV channel and is auto-started when the user switches to that related TV channel. This is normally a fixed URL which doesn't change over the run-time of the application.

For dynamic events there is the possibility to use stream events which are part of the DSM-CC system (Digital storage media command and control) available in HbbTV consisting of small data packets that can be transmitted synchronously with the programme material. The purpose of the stream events is to send triggers to the client application. This can be used to having a timed link between the broadcast and the broadband world, e.g. for synchronising devices or just to trigger events at a certain time. As discussed previously, MPAT will provide number of methods to create the stream events. For example, an event schedule can be set up, where the author lists the triggers relevant to the application with the help of relative time-stamps. Furthermore, there is the potential to create stream events during the run-time of the applications via a manual live-trigger, e.g. by an editor. Both options have to be processed by the stream event inserter, a component in the external backend of MPAT.

While the web server feeds the frontend client with the broadband information, the TV-Playout API handles the broadcast side of the playout system. The MPAT consortium decided to define a generic interface, including an external AIT configuration component and the stream event inserter. For this purpose MPAT enables the connection of various TV modulators. Those modulators are needed to generate the transport stream, where the relevant application information and app triggers are inserted into the broadcast signal.

Ultimately MPAT supports two different ways of broadcast deployment. Firstly, finalised apps need to be tested in an appropriate environment, but this can't be done in the scope of a production playout of the broadcaster. Therefore a possibility to test the authored apps with various test systems has to be facilitated, which will be supported by a generic Playout API. As a first prototypical implementation, it has been decided to use IRT's BRAHMS solution for this matter. Secondly the apps have to be ingested in the production system of the broadcaster for final playout via an appropriate interface.


31

Glossary

AIT Application Information Table

AC Asset Converter

API Application Programming Interface

CDN Content Delivery Network

CMS Content-Management -System

CSS Cascading Style Sheets

DIAL Discovery and Launch

DSM-CC Digital storage media command and control

DVB Digital Video Broadcasting

HbbTV Hybrid broadcast broadband TV

JSON JavaScript Object Notation

MPEG Moving Picture Experts Group

REST Representational State Transfer

TS Transport Stream

VoD Video-on-Demand

WP WordPress

system architecture and api documentation...

Documents