bas engel royal philips / consumer electronics division celf embedded linux conference 2007 april...

Bas EngelRoyal Philips / Consumer Electronics DivisionCELF Embedded Linux Conference 2007April 17, 2007

SPACESPlit Application architeCturE

Enabling predictable SW integration

2Royal Philips / Consumer Electronics Division, Engel, April 17, 2007

Introduction


SW in TV

0%

20%

40%

60%

80%

100%

2005 2006 2007

CE

3rd Party

NXP

Software size evolution Software partitioning

• Total SW size follows Moore’s law

• Philips part decreasing (not just relative)

• 3rd party involvement increasing


Some rationales on 3rd party SW

• Cost, however – balance NRE and royalties vs. own development cost

• TTM– No time to build up competence in own team– “Off the shelf solution available”– Critical competence available at supplier side

Somebody else makes a business out of supplying

common features

Somebody else makes a business out of supplying

common features


Traditional approach: integrate

MIPS (Linux 2.x)

process

Philips

3rd party


Integration nightmare

• Integration has it’s advantages– Optimized resource usage– Better viewing experience due to integral architecture approach

• Integration pitfall– High effort due to increased complexity– Longer TTM as there is no little change– System validation requires global big bang approach– No independent lifecycle, no distributed

integration cycle



cbmheg cegs

osinfraVprocavfeaprocdec

tv520avi

avplf

txsvc

juice

hysvc

gfxmgr

txplf

tv520avi

psfactory

pssvc

pstvapps

Vproc

osinfra

osinfra osinfra

tv520avi

UIEase of use

Service, Power, Storage

Broadcast

Display, sound, connection PQ

Infra

Audio, Video

MIPS (Linux 2.x)

process

Philips

3rd party



cbmheg cegs


tv520avi

avplf

txsvc

juice

hysvc

gfxmgr

txplf

tv520avi

psfactory

pssvc

pstvapps

Vproc

osinfra

osinfra osinfra

tv520avi

UIEase of use


Broadcast


Infra

Audio, Video

MIPS (Linux 2.x)

process

Philips

3rd party


• We have to create an environment in which 3rd parties can integrate their software

– Without deep system knowledge• We have to enable predictable integration

– Preventing spaghetti SW• We must cater for sharing critical resources

– AVG Platform– General purpose infrastructure

The challenge

Create a Win-Win atmosphere with 3rd parties Create a Win-Win atmosphere with 3rd parties


Design objectives

• Fast and predictable integration of system extensions– Avoid an extensive (re)validation cycle– Enable convincing module test opportunities– Enable PC based testing

• Orthogonal 3rd party SW integration– Leverage standard Linux infrastructure– Multi client usage of platform resources

• Manage independently deployable building blocks– Limited building block correlation– Cater for extensions without the need to know all the details

• Enhanced execution architecture– Independent application lifecycle

• Enable true Multi window architecture– For Video and Graphics– Preserving application orthogonality


The mental model: SW bolt-ons

TV Platform

Feat4Feat1 Feat2 Feat3TV

Middleware

MIPS (Linux 2.x)

process



cbmheg cegs


tv520avi

avplf

txsvc

juice

hysvc

gfxmgr

txplf

tv520avi

psfactory

pssvc

pstvapps

Vproc

osinfra

osinfra osinfra

tv520avi

UIEase of use


Broadcast


Infra

Audio, Video

MIPS (Linux 2.x)

process

Philips

3rd party


SPACE


Introducing SPACE

Linux 2.x

TV platform

TVMiddleware

ApplicationManager

FeatN

Feat1

Application

Platform API

Have orthogonal applications• The resources in the system are

explicitly and centrally managed• The client applications are system

context unaware• The lifecycle, focus and visual layout of

the client applications is centrally managed


Application Manager

• Application lifecycle management– Starting, stopping applications

based on remote control keys– Knows the system requirements

of all applications• Focus handling

– Determines the active window, reacts to user request

– Manages the application requirements to the AV resources

• Layout management– Determines how the applications

are presented


Managing resources

• There are implicitly managed resources by the kernel– TCP, flash, USB– Memory allocation

• There are explicitly managed resources– AV platform resources– Memory and CPU resources

• The explicitly managed resources– Have a statically defined execution behavior– Can by design limit the parallelism in the system– Require an explicitly resource controlled system


Resource Controlled System

• Application Manager (amApp) – Knows the resource dependencies of any client application

• Statically by design

• The platform application (plfApp)– Has a static model of resource sharing

• By design resources are multi/single client

• Every application – Must be resource aware

• Meaning the resource is or is not reserved for them


Multi Client Management

tvApp

Linux 2.x

plfAppamApp

SetFrequency (tuner)

otherApp


ResourceOwner (tvApp)


Resource Groups

• There are different resource groups offered by the platform– Front-end, demux, decode, viewing mode, source selection– To allow multiple applications to access different parts of the

system– Every resource group has a static set of interfaces

• The Application Manager is the main Resource controller– It designates resource groups to clients– By informing the platform application whom to grant access



tvApp

Linux 2.x

plfAppamApp


otherApp





tvApp

Linux 2.x

plfAppamApp


otherApp




The platform API

• The Philips platform API– Leverages as much as possible the current APIs– Enables the resource sharing model– Is supplier independent

• AV Interface definition– Analog interfaces: Philips’ Analog TV API– Digital interfaces: NXP’s Digital TV API

• AV Interface instance– Simple header files and libraries– Tool to generate proxy/stub code

• Graphics interface definition– DirectFB

? ??


Introducing SPACE

Linux 2.x

TV platform

TVMiddleware

ApplicationManager

FeatN

Feat1

Application

Platform API






Introducing SPACE

Linux 2.x

TV platform

TVMiddleware

ApplicationManager

FeatN

Feat1

Application

Platform API






General Connection Management Concept

• Connection Management is split up in– Destination setup

• Output: FullScreen, PIP– Source setup

• Input: HDMI, Tuner

• Connection Management is distributed– Application Manager is responsible for

• Destination setup• Client application lifecycle

– Client applications are responsible for• Source setup

• Client applications are destination unaware• Application Manager is source unaware


Video Connection Management

plfApp

tvApp

App3

App2

source

amApp

destination

Configure

Select

Configure


Video Connection Management

plfApp

tvApp

App3

App2

source

amApp

destination

Configure

Select

Configure


Example

tvApp

Linux 2.x

plfAppamApp

2 eventDestination(tvApp)

1 SetDestinationFullScreen(tvApp)

5 eventOnSourceSelected()

4 SelectTuner

3 SetSource(tuner)6 Program Freq, PID, …


Border Windows and Video Windows

• Border windows are linked to video windows– Every application controlling video creates a border window– Application manager controls visibility of such a couple

• Visible border windows are equal to the visible video windows– There can be multiple border windows for 1 video window– The border window determines the position of the video window

• DirectFB support this via the concept of Input Only Windows– Geometry (positioning) is still available to allow focus management– No client buffer is required, no scaling is done

Border

Video


Graphics Connection Management

DirectFB infrastructure

source

amApp

destination

config

ureDraw

SelecttvApp

App1

App2


Window Management responsibilities

tvApp

Ap

p1 Ap

p2

create draw

amApp

focusLayoutstart/kill

tvApp

Ap

p1 Ap

p2

Size


Traditional DirectFB Window manager

WM Module

App App

SHM

DirectFB API

• Pluggable WM API– Move, Resize, Raise, Lower, Hide, Show Windows– Process Input, Handle Focus and Grabbing, Dispatch Events– Full Cursor implementation including Show, Hide, Reshape, Move– Composition of the screen or parts, e.g. triggered by

• Flip() on a window surface• Move, Raise, Show etc.

– 2D composition• Background, Windows, Cursor

• Default WM module– Just follow application requests– Basic focus handling


SaWMan

• Shared Application & Window Manager– New DirectFB window manager

• On its own like default window manager implementation– Default is maintained to preserve backwards compatibility

• Required was a more controlled behavior– Add/Remove windows, can modify initial configuration– Configuration requests (move, resize, opacity), modify or reject– Filter input events before being processed by SaWMan– Completely overrule window layout, enable focus borders etc.

• Application Manager can hook into most of the API flow

SaWMan Module

App App

DirectFB API

App

SaWMan API

Application Manager

SHMRPC to AM


Window Management

Message

Application

Border

Child abducted San Jose

Z-orderLayers

low

med

high

FocusmgtClients

VideoHW

Child abducted San Jose


• If we need fast rendering (>10FPS) and no HW acceleration available– We must use direct surface drawing– To achieve >25 FPS without SW scaling overhead

• DirectFB must allow Layers to be mixed across HW Surfaces– Some Layers can have SW scaling, others may not

DirectFB layer mapping on HW

Message

Application

Border

Z-orderLayers

low

med

high

Surface

Video HW

GFX1

GFX1

GFX1

Video0

Message

Application

Border

Z-orderLayers

low

med

high

Surface

Video HW

GFX1

GFX1

GFX0

Video0

Versatile

Performance

Video

Graphics

Legend

Option 1 Option 2


Window Association

• Windows can be associated to another– Meaning identical position and size

• Focus only to associated group– Focus only to top window, not associated window

• Example use-case: teletext + menu– Teletext and menu of teletext separate windows

ClientCreateAssociative

Draw Flipwindow

Draw

time


Pixel Alignment• plfApp can crop video

– Eg to remove DNM border artifacts• plfApp can scale video within border window

– Eg to set the scaling to 4:3 or 16:9• Any client application can indicate ‘pixel

alignment’– To assure that the graphics is pixel

accurately positioned on top of the video• DirectFB caters for the alignment

– plfApp configures the scale/crop factors

• Other application is pan/zoom– Any application can use the SW scaling

Crop &Scale

VideoAligned

Crop valueCrop &Scale


Road to success


Planned extensions to DirectFB

• SawMan– Shared Application Window Manager

• Enhanced SW up/down scaling for windows– With dynamic layer reconfiguration and color key conversion

• FusionDale & FusionIPC– Applied Fusion for Event handling and IPC

• Memory usage optimization– Shared Memory Heap

• Window Association– Link windows together for size and location

• Window mapping on HW layers– Including layer specific configurations Pixel alignment

• Extended Key handling– Sending keys to windows and return key not used

• Audio Nodes– General ID for audio resources

• Multi processor SPACE– Enable multiple control processors to host client applications

Pu

blish

ed

To

be

pu

blish

ed

Pla

n


Main conclusion

• The integration nightmare can become manageable– Using an open standard based multi application approach

• DirectFB, Linux– With a clear orthogonal view

• No dependencies between applications– Adding the necessary components for success

• SPACE, SaWMan

• SPACE– Is the architectural concept for multi application systems

• Coping with the resource constraints in CE devices

• SaWMan – Enables application lifecycle management, focus handling,

and window management• Is fully available via the DirectFB mainline


Introducing SPACE

Linux 2.x

TV platform

TVMiddleware

ApplicationManager

FeatN

Feat1

Application

AVG API






Introducing SPACE

Linux 2.x

TV platform

TVMiddleware

ApplicationManager

FeatN

Feat1

Application

AVG API






SaWMan

• Shared Application & Window Manager– On its own like default window manager implementation

• Default is maintained to preserve backwards compatibility– Required was a more controlled behavior

• Add/Remove windows, can modify initial configuration• Configuration requests (move, resize, opacity), modify or reject• Filter input events before being processed by SaWMan• Completely overrule window layout, enable focus borders etc.

– Application Manager can hook into most of the API flow

SaWMan Module

App App

DirectFB API

App

SaWMan API

Application Manager

SHMRPC to AM

DirectFB

Application Manager

sawman

Key Routing

key injection

window creation

window destruction

process creation

process destruction

set focus

hide window

layout information

Available Applications

Special Keys

Application requirements

Layout selection

window changes

key grabbing

key collecting

show window


SaWMan

• Shared Application & Window Manager– On its own like default window manager implementation

• Default is maintained to preserve backwards compatibility– Required was a more controlled behavior

• Add/Remove windows, can modify initial configuration• Configuration requests (move, resize, opacity), modify or reject• Filter input events before being processed by SaWMan• Completely overrule window layout, enable focus borders etc.

– Application Manager can hook into most of the API flow

SaWMan Module

App App

DirectFB API

App

SaWMan API

Application Manager

SHMRPC to AM

DirectFB

Application Manager

sawman

Key Routing

key injection

window creation

window destruction

process creation

process destruction

set focus

hide window

layout information

Available Applications

Special Keys

Application requirements

Layout selection

window changes

key grabbing

key collecting

show window






The challenge







The challenge



Demo: SPACE

bas engel royal philips / consumer electronics division celf embedded linux conference 2007 april...

Documents

moores law philips

total sw size

integratemips linux

party swcost

managedthe client applications

little changesystem

block correlationcater

supplier sidesomebody