multimedia presentation and delivery ch.13 principles of multimedia database systems. v.s....

Multimedia Presentation and Multimedia Presentation and DeliveryDelivery

Ch.13 Ch.13

Principles of Multimedia Database Principles of Multimedia Database Systems. V.S. Subrahmanian, 1998Systems. V.S. Subrahmanian, 1998

MM Presentation & DeliveryMM Presentation & Delivery 22

Presentation & RetrievalPresentation & Retrieval When creating a multimedia presentation, three basic questions When creating a multimedia presentation, three basic questions

must be answeredmust be answered• WhatWhat objects should be included in the presentation ? objects should be included in the presentation ? • WhenWhen should these objects to be presented to the user ? should these objects to be presented to the user ? • WhereWhere should the objects appear on the screen ? should the objects appear on the screen ?

These questions must be answered by the individual who creates These questions must be answered by the individual who creates the presentation (i.e., author) the presentation (i.e., author)

Once these have been specified, the Presentation Engine must Once these have been specified, the Presentation Engine must generate a Retrieval Plan like:generate a Retrieval Plan like:• when the objects need to be presented when the objects need to be presented • bandwidth limitation on the network bandwidth limitation on the network • resource (load, buffer) limitation on the server & client resource (load, buffer) limitation on the server & client • mismatches between delivery rate and client consumption rate.mismatches between delivery rate and client consumption rate.


Interaction Between Presentation & Retrieval SchedulesInteraction Between Presentation & Retrieval Schedules


Creating a PresentationCreating a Presentation

Suppose objects oSuppose objects o11,...,o,...,onn. are to be presented.. are to be presented. Temporal constraints specify how the objects should be Temporal constraints specify how the objects should be

laid out in time. Example:laid out in time. Example:• The presentation of objects 1 and 2 must start at the same The presentation of objects 1 and 2 must start at the same

time.time.• The termination of objects 2 and 3 must occur at the same The termination of objects 2 and 3 must occur at the same

time.time.• Object 3 must start at the time object 1 ends.Object 3 must start at the time object 1 ends.

Spatial constraints specify how the objects should be laid Spatial constraints specify how the objects should be laid out in space. Example:out in space. Example:• Object 1 must be to the left of object 2.Object 1 must be to the left of object 2.• Object 1 must be above object 3.Object 1 must be above object 3.


Seven Possible Temporal RelationshipsSeven Possible Temporal Relationships


Constraint LanguageConstraint Language ConstantsConstants: Every integer (positive and negative) is a constant.: Every integer (positive and negative) is a constant. VariablesVariables: Associated with each o: Associated with each oii are two integer variables, s are two integer variables, sii

(denoting the "start" of o(denoting the "start" of oii) and e) and eii (denoting the "end" of o (denoting the "end" of oii)) Elementary TermsElementary Terms: Elementary terms are defined inductively : Elementary terms are defined inductively

as follows:as follows:• Every constant is an elementary termEvery constant is an elementary term• Every variable is an elementary termEvery variable is an elementary term

Difference ConstraintDifference Constraint: If tl, t2 are elementary terms, and c is a : If tl, t2 are elementary terms, and c is a Constant, thenConstant, then• tl - t2 < Ctl - t2 < C

is a difference constraintis a difference constraint


Examples of Difference ConstraintsExamples of Difference Constraints

e1 - s1 e1 - s1 10. 10.

This constraint says that object oThis constraint says that object o11 must end within 10 time must end within 10 time

units of the time its presentation starts.units of the time its presentation starts. s2 - e1 s2 - e1 0; e1 - s2 0; e1 - s2 0 0

These two constraints jointly state that object oThese two constraints jointly state that object o11's 's presentation starts as soon as object opresentation starts as soon as object o11's presentation ends, 's presentation ends, i.e., s2 - e1 = 0i.e., s2 - e1 = 0

s2 - e1 s2 - e1 3 3

This constraint says that object oThis constraint says that object o22's presentation starts within 's presentation starts within 3 time units of the end of the presentation of object o3 time units of the end of the presentation of object o11..


A Definition of Temporal PresentationA Definition of Temporal Presentation A temporal presentation is a pair TP = (O, DC) where O is a A temporal presentation is a pair TP = (O, DC) where O is a

finite set of objects, and DC is a finite set of difference finite set of objects, and DC is a finite set of difference constraints in the constraint language generated by O.constraints in the constraint language generated by O.

Suppose DC is a set of difference constraints over O of n Suppose DC is a set of difference constraints over O of n virtual objects. A solution of DC is an assignment of virtual objects. A solution of DC is an assignment of integers to each of the variables, s1, ... sn, e1, ... en, which integers to each of the variables, s1, ... sn, e1, ... en, which makes all the constraints true.makes all the constraints true.

E.g.,E.g.,

Sets of constraints can have no, one, multiple solutions.Sets of constraints can have no, one, multiple solutions.


Possible solutions:Possible solutions:

A temporal presentation TP = (O, DC) is feasible iff the A temporal presentation TP = (O, DC) is feasible iff the set, DC, of difference constraints, has a solution set, DC, of difference constraints, has a solution , I.e., a , I.e., a feasible schedule for TP.feasible schedule for TP.


Definitions about the scheduleDefinitions about the schedule The start and end of The start and end of , denoted start(, denoted start() and end() and end(), are ), are

defined to be:defined to be:• start(start() = min({) = min({(si) (si) II1 1 i i n}). n}).• end(end() = max({) = max({(ei) (ei) II1 1 i i n}). n}).

We are assuming that the constraints {si - ei We are assuming that the constraints {si - ei 0|1 0|1 i i n} are included in DC, i.e., a default DC.n} are included in DC, i.e., a default DC.

For our example:For our example:


How to create a presentation schedule?How to create a presentation schedule?

We are facing a constraint satisfaction problem which can We are facing a constraint satisfaction problem which can be solved by linear programming algorithms like simplex be solved by linear programming algorithms like simplex method.method.

There is another classic algorithm called the Bellman Ford There is another classic algorithm called the Bellman Ford Algorithm whichAlgorithm which• takes as input, a set S of difference constraints.takes as input, a set S of difference constraints.• Convert S to a graph.Convert S to a graph.• S has a solution iff the graph has no negative cycles.S has a solution iff the graph has no negative cycles.• A negative cycle is a cycle whose edges sum up to a A negative cycle is a cycle whose edges sum up to a

negative number.negative number.


An example:An example:Given a set S of DC’s:Given a set S of DC’s:

Form a graph for S:Form a graph for S:


An example: (cont.)An example: (cont.)

Compute the shortest/cheapest pathsCompute the shortest/cheapest paths

If no negative cycles (i.e., If no negative cycles (i.e.,

costs are not negative), costs are not negative),

we have we have


More about Bellman-Ford AlgorithmMore about Bellman-Ford Algorithm Let the cost of a path be the sum of the costs of edges along Let the cost of a path be the sum of the costs of edges along

the path.the path. A cycle in graph G is a sequence of nodes A cycle in graph G is a sequence of nodes vv1 1 , … , v, … , vkk such that such that

vv1 1 = v= vkk and and 1 1 j < k, ( j < k, (vvj j , v, vj+1 j+1 ) ) is an edge in the graph.is an edge in the graph.

vv1 1 , … , v, … , vkk is said to be a negative cycle in G iff it (total) cost is is said to be a negative cycle in G iff it (total) cost is a negative numbera negative number

A set of difference constraints DC has no solution iff GA set of difference constraints DC has no solution iff GDCDC has a negative cycle.has a negative cycle.

Suppose a graph has no negative cycles. Then the cost of Suppose a graph has no negative cycles. Then the cost of the shortest/cheapest path from the start node to that node (si the shortest/cheapest path from the start node to that node (si or ei) directly allows us to get a solution to the original set of or ei) directly allows us to get a solution to the original set of difference constraints.difference constraints.


Algorithm ImplementationAlgorithm Implementation

The Bellman Ford algorithm associates with each node N in The Bellman Ford algorithm associates with each node N in GGDCDC, the following two fields:, the following two fields:

BestvalBestval: This specifies the cheapest path from the start : This specifies the cheapest path from the start node to the node N, that has been discovered thus farnode to the node N, that has been discovered thus far

BestparBestpar: : Bestpar(N)Bestpar(N) specifies the immediate predecessor of specifies the immediate predecessor of node N along the best path from the start node to node N, node N along the best path from the start node to node N, that we have found thus farthat we have found thus far

Then,Then, Execute Algorithm 13.3 (which includes Algorithm 13.1, Execute Algorithm 13.3 (which includes Algorithm 13.1,

13.2) in the text.13.2) in the text.


A Broader Issue of MM Presentation: A Broader Issue of MM Presentation: Multimedia SynchronizationMultimedia Synchronization

Attempts to ensure the desired temporal ordering among a set of Attempts to ensure the desired temporal ordering among a set of MM objects in a multimedia scenario, where “MM scenario” MM objects in a multimedia scenario, where “MM scenario” denotes the temporal semantics of a MM sessiondenotes the temporal semantics of a MM session

The goals of MM synchronization mechanism are to preserve the The goals of MM synchronization mechanism are to preserve the playback continuity of media frames within single continuous playback continuity of media frames within single continuous media and the desired temporal dependencies among multiple media and the desired temporal dependencies among multiple related data objects, given that user interactions are allowed.related data objects, given that user interactions are allowed.

Temporal dependency can be either implicit (lip synchronization) Temporal dependency can be either implicit (lip synchronization) or explicit (a slide show)or explicit (a slide show)

MM synchronization is an important set of QoS parametersMM synchronization is an important set of QoS parameters MM synch. differs from traditional synch. mechanisms such as MM synch. differs from traditional synch. mechanisms such as

monitor and semaphore in that it imposes real-time constraints monitor and semaphore in that it imposes real-time constraints (deadlines) on events, and differs from real-time systems in that (deadlines) on events, and differs from real-time systems in that such deadlines are predetermined, periodic, and softsuch deadlines are predetermined, periodic, and soft


Three Levels of MM SynchronizationThree Levels of MM Synchronization

Intramedia synchronizationIntramedia synchronization also known as intrastream or serial synchronizationalso known as intrastream or serial synchronization preserves the temporal relationship between consecutive preserves the temporal relationship between consecutive

frames and the continuity of playback within a single media frames and the continuity of playback within a single media streamstream

the basic idea is to smooth the delay jitters experienced in the the basic idea is to smooth the delay jitters experienced in the storage system, network, and host systemstorage system, network, and host system

Intermedia synchronizationIntermedia synchronization also known as interstream or parallel synchronizationalso known as interstream or parallel synchronization coordinates the different media streams to achieve the desired coordinates the different media streams to achieve the desired

temporal relationships among themtemporal relationships among them


Intra and Intermedia Synchronization RequirementsIntra and Intermedia Synchronization Requirements


Three Levels of MM Synchronization (cont.)Three Levels of MM Synchronization (cont.)

Interparty synchronizationInterparty synchronization also known as event synchronizationalso known as event synchronization maintains the intramedia and/or intermedia synch. among maintains the intramedia and/or intermedia synch. among

different participants at distributed locationsdifferent participants at distributed locations e.g., remote learning and video conferencinge.g., remote learning and video conferencing


Temporal Synchronization MechanismsTemporal Synchronization Mechanisms(in a distributed environment)(in a distributed environment)

Synchronization mechanism renders the synchronization Synchronization mechanism renders the synchronization specification into a set of playback schedules and retrieval specification into a set of playback schedules and retrieval schedule and enforces the playback deadlines by taking into schedule and enforces the playback deadlines by taking into considerations of the network characteristicsconsiderations of the network characteristics

e.g. to display a MM object at time s (i.e., the playback e.g. to display a MM object at time s (i.e., the playback deadline), the presentation site sends a request to the deadline), the presentation site sends a request to the corresponding media server at request time r, which is R time corresponding media server at request time r, which is R time units prior to s. R(=s-r)is the response time to account for units prior to s. R(=s-r)is the response time to account for possible experienced end-to-end delays and packet losses.possible experienced end-to-end delays and packet losses.


Temporal Synchronization Mechanisms: delayTemporal Synchronization Mechanisms: delay

For discrete object, the playback deadline can be met when R For discrete object, the playback deadline can be met when R the response time is larger than the total delay (server the response time is larger than the total delay (server ddserser + +

network network ddnetnet + host delay + host delay ddhosthost))

e.g. to retrieve an image of size 1.3MB from a remote site with e.g. to retrieve an image of size 1.3MB from a remote site with ddserser=50=50s connected by a link of 100Mbps bandwidth, we should s connected by a link of 100Mbps bandwidth, we should

have the response timehave the response time

RR 50 50s+1.3x8Mbits/100Mbits+ s+1.3x8Mbits/100Mbits+ ddhosthost

ddhost host is normally negligible. is normally negligible.


Temporal Synchronization Mechanisms: delay jitterTemporal Synchronization Mechanisms: delay jitter

For a continuous media object, each media frame/packet may For a continuous media object, each media frame/packet may experience different delay and such intrastream synchronization can experience different delay and such intrastream synchronization can be achieved by delaying the the playback deadline of the first media be achieved by delaying the the playback deadline of the first media frame by frame by ddsynsyn, i.e.,, i.e.,

which is the difference between the maximum and minimum network which is the difference between the maximum and minimum network delays of stream i.delays of stream i.

Therefore, the response time should be at least (server Therefore, the response time should be at least (server ddserser + network + network

ddnetnet + intramedia synchronization + intramedia synchronization ddsynsyn + host delay + host delay ddhosthost)) e.g., to retrieve a video clip of fixed frame size 40kB from a remote site e.g., to retrieve a video clip of fixed frame size 40kB from a remote site

with with ddserser=50=50s connected by a link of 100Mbps bandwidth and 10s connected by a link of 100Mbps bandwidth and 10s s

intramedia, we should have response timeintramedia, we should have response time

R R 50 50s + 40x8kbits/100Mbits + 10s + 40x8kbits/100Mbits + 10s + s + ddhosthost

iisyn ddd minmax

MM SynchronizationMM Synchronization 2323

More on Synchronization RequirementsMore on Synchronization Requirements Synchronization can be measured by four major parameters: delay, Synchronization can be measured by four major parameters: delay,

delay jitter (intramedia skew), intermedia skew, and tolerable errordelay jitter (intramedia skew), intermedia skew, and tolerable error Delay measures end-to-end delay or response timeDelay measures end-to-end delay or response time Delay jitter measures delay variations of continuous mediaDelay jitter measures delay variations of continuous media Intermedia skew measures the time shift between related media from Intermedia skew measures the time shift between related media from

the desired temporal relationship, e.g., lip synchronization requires the desired temporal relationship, e.g., lip synchronization requires intermedia skew within ±80ms, stereo audio (audio-audio intermedia skew within ±80ms, stereo audio (audio-audio synchronization) requires ±11synchronization) requires ±11s, video-text synchronization requires s, video-text synchronization requires ±240ms ±240ms

Tolerable error measures the allowable bit error rate and packet error Tolerable error measures the allowable bit error rate and packet error rate for a particular stream in a particular application, e.g., bit error rate for a particular stream in a particular application, e.g., bit error rate should be smaller than 0.1 (for uncompressed voice), 0.01 (for rate should be smaller than 0.1 (for uncompressed voice), 0.01 (for uncompressed TV-quality video), 0.0001 (for uncompressed image)uncompressed TV-quality video), 0.0001 (for uncompressed image)


Synchronization Specifications & Synchronization Specifications & MechanismsMechanisms

Synchronization can be refer to time (our focus here), space, and Synchronization can be refer to time (our focus here), space, and content. content.

General MM synchronization can be divided into two levels: General MM synchronization can be divided into two levels: • (temporal) synchronization specification(temporal) synchronization specification (upper level) (upper level) • (temporal) synchronization mechanism(temporal) synchronization mechanism (lower level) ! (lower level) !

Synchronization specificationSynchronization specification models the temporal relationships models the temporal relationships among the set of related abstract MM objects (e.g., video clip, among the set of related abstract MM objects (e.g., video clip, text) of a presentation scenario text) of a presentation scenario

Synchronization mechanismSynchronization mechanism translates the temporal translates the temporal synchronization specification into the desired presentation synchronization specification into the desired presentation scheduling sequences and enforces the playback deadlines scheduling sequences and enforces the playback deadlines despite the indeterministic delays due to the server, the network despite the indeterministic delays due to the server, the network and the host systemand the host system


Specification by Scripts Specification by Scripts

(Language-Based Approach)(Language-Based Approach) A script uses a structural language (concurrent programming language) to A script uses a structural language (concurrent programming language) to

specify the temporal relationships, e.g., Communication Sequential specify the temporal relationships, e.g., Communication Sequential Processing (CSP) schemeProcessing (CSP) scheme

Scripts are powerful and can be used to specify all types of relationships Scripts are powerful and can be used to specify all types of relationships and user interactionsand user interactionsScript slide-show Script slide-show

{{

ParallelParallel

Display picture1.jpg for 5 secondsDisplay picture1.jpg for 5 seconds

Display audio1.au for 5 secondsDisplay audio1.au for 5 seconds

ParallelParallel


Display audio2.au for 7 secondsDisplay audio2.au for 7 seconds

ParallelParallel


Display audio3.au for 10 seconds Display audio3.au for 10 seconds

}}


Specification by Scripts (cont.)Specification by Scripts (cont.)

The first line specifies the name of the scriptThe first line specifies the name of the script Parallel, Display, For, Parallel, Display, For, andand Seconds Seconds are keywords of the script are keywords of the script A parallel structure completes when all statements within it A parallel structure completes when all statements within it

complete; so in the second parallel structure, the audio complete; so in the second parallel structure, the audio continues to play for 1 sec after the display of picture 2 has continues to play for 1 sec after the display of picture 2 has ended (A starts B).ended (A starts B).

Advantage of using scripts: they can be directly lead to Advantage of using scripts: they can be directly lead to implementationimplementation

Disadvantage: they are hard to conceptually visualize and are Disadvantage: they are hard to conceptually visualize and are difficult to verifydifficult to verify


Specification by Time LinesSpecification by Time Lines

In this specification, the presentation time of each stream is In this specification, the presentation time of each stream is specified relative to a common clock.specified relative to a common clock.

Streams are treated independently.Streams are treated independently. Synchronization is achieved by ensuring that each stream is Synchronization is achieved by ensuring that each stream is

presented at its specified time relative to the common clockpresented at its specified time relative to the common clock Advantages: simple and easy to use and implement because Advantages: simple and easy to use and implement because

streams are treated independently during executionstreams are treated independently during execution Disadvantages: cannot specify user interaction or streams with Disadvantages: cannot specify user interaction or streams with

unpredictable durationunpredictable duration

refer to Fig.9.2, p.279 of reference materialrefer to Fig.9.2, p.279 of reference material


Specification by GraphsSpecification by Graphs SynchronizationSynchronization can be specified or modeled by graph, in can be specified or modeled by graph, in

particular, object composition petri nets (OCPN)particular, object composition petri nets (OCPN) The basic idea is to represent various components of MM The basic idea is to represent various components of MM

objects as objects as placesplaces and describe their inter-relations in the form of and describe their inter-relations in the form of transitionstransitions

It is capable to explicitly capture all the necessary temporal It is capable to explicitly capture all the necessary temporal relations, and to provide simulation of presentation in both the relations, and to provide simulation of presentation in both the forward and reverse directions (better user interaction)forward and reverse directions (better user interaction)

Each place in the model represents the play-out of a MM object Each place in the model represents the play-out of a MM object while transitions represent synchronization pointswhile transitions represent synchronization points

It specifies exact presentation time play-out semantics, which is It specifies exact presentation time play-out semantics, which is useful in real-time schedulinguseful in real-time scheduling


Specification by OCPNSpecification by OCPN

The slide show example:The slide show example:

Another example:Another example:

multimedia presentation and delivery ch.13 principles of multimedia database systems. v.s....

Documents