cs 414 – multimedia systems design lecture 25 – synchronization issues
DESCRIPTION
CS 414 – Multimedia Systems Design Lecture 25 – Synchronization Issues. Klara Nahrstedt Spring 2012. Administrative. MP2 posted MP2 Deadline – April 7, Saturday, 5pm. Internet Multimedia Protocol Stack. Media encaps (H.264, MPEG-4). Synchronization Service. DASH. APPLICATION. RTSP. - PowerPoint PPT PresentationTRANSCRIPT
CS 414 - Spring 2012
CS 414 – Multimedia Systems Design Lecture 25 – Synchronization Issues
Klara NahrstedtSpring 2012
Administrative MP2 posted MP2 Deadline – April 7, Saturday, 5pm.
CS 414 - Spring 2012
Internet Multimedia Protocol Stack
CS 414 - Spring 2012
AAL3/4
IP Version 4, IP Version 6
UDP
Media encaps(H.264, MPEG-4)
RTP
ATM/Fiber Optics Ethernet/WiFi
TCP
SIP RTSP RSVP RTCP
AAL5
KE
RN
EL
AP
PLIC
ATIO
N
Layer 4(Transport)
Layer 3(Network)
Layer 2(Link/MAC)
Layer 5(Session)
MPLS
DCCP
DASH
HTTP
Synchronization Service
Outline
Synchronization in Distributed Environments
Clock Synchronization Synchronization Reference Model
Media, Stream, Object, Specification Levels
CS 414 - Spring 2012
Synchronization in Distributed Environments Information of synchronization must be transmitted
with audio and video streams, so that receiver(s) can synchronize streams
Sync information can be delivered before start of presentation (used by synthetic synchronization) Advantage: simple implementation Disadvantage: presentation delay
Sync information can be delivered using separate sync channel - out-band (used by live synchronization) Advantage: no additional presentation delay Disadvantage: additional channel needed
CS 414 - Spring 2012
Sync in Distributed Environments
Sync information can be delivered using multiplexed data streams - in-band sync Advantage: related sync information is
delivered together with media unitsDisadvantage: difficult to use for multiple
sources
CS 414 - Spring 2012
Location of Sync Operation Sync media objects by combining objects into
new media object Sync operation placed at sink
Demand on bandwidth is larger because additional sync operations must be transported
Sync operation placed at sourceDemand on bandwidth smaller because streams
are multiplexed according to sync requirements
CS 414 - Spring 2012
Clock Synchronization Sync accuracy depends on clocks
at source and sink nodes Ta = Tav – Nla – Oa
Tv = Tav – Nlv – Ov End-to-end delay
Nla = EEDa = Tav-Ta-Oa
Nlv=EEDv = Tav-Tv-Ov
EEDa = (Ta1-Ta2)/2 NTP (Network Time Protocol )
CS 414 - Spring 2012
AV Skew Calculation Select synchronization point in AV streams in
terms of frame number for each stream Execute sync operation at sync points1. Measure
Arrival time of sync audio packet TsyncA
Arrival time of sync video packet TsyncV
2. Calculate: AVSkew = TsyncA – TsyncV
3. Alternative measure and calculate TsyncA = TsendA + EEDA; TsyncV = TsendV + EEDV
EED = RTT/2 = (Trecv – Tsend)/2 CS 414 - Spring 2012
Network Time Protocol Protocol to sync clocks of computer systems
over packet-switched, variable –latency data networks Uses UDP port 123 Designed to resist effects of variable latency (jitter buffer) Designed in 1985 by Dave Mills at U. Delaware Can achieve accuracy of 200 µsec Based on Marzullo Algorithm
CS 414 - Spring 2012
Marzullo’s algorithm (1984)(intersection algorithm)
Agreement protocol for estimating accurate time from a number of noisy time sources
If we have estimates 10 2, 12 1, 11 1, then
interval intersection is 11.5 0.5
If some intervals don’t intersect, consider intersection of majority of intervals
CS 414 - Spring 2012
Source: wikipedia
Clock strata NTP uses hierarchical system of
“clock strata” Stratum levels define distance
from reference clock and exist to prevent cycles in hierarchy Stratum 0
devices are atomic clocks, GPS clocks, radio clocks
Stratum1 computers attached to stratum0 devices Act as servers for timing requests from
Stratum 2 servers via NTP Stratum2
(similar to Stratum1, but they also have peering relation to other stratum2 servers CS 414 - Spring 2012
Precision Time Protocol (PTP) High Precision time protocol (original IEEE 1588-2002)
for synchronization used in measurement and control systems residing on a local area network
Accuracy in the sub-microseconds PTPv2 – IEEE 1588-2008 improves accuracy, precision
and robustness PTP serves a niche not well served by NTP and GPS
it achieves accuracy beyond those attainable using NTP;
It is cheaper than GPS It is accessible to receivers for which GPS signals are
inaccessible. CS 414 - Spring 2012
PTP Details IEEE 1588 standards describe a hierarchical master-
slave architecture for clock distribution Ordinary clock – source for root timing reference, called
‘grandmaster’ Boundary clock – intermediate synchronization bridge from one
network segment to another – used by synchronization master to sync clocks within a network segment
PTP messages use IP-multicast over UDP (IPv4 or IPv6)
Selection algorithm of best clock source uses properties such as quality, accuracy, variance, …
CS 414 - Spring 2012
PTP Synchronization After selection of master source of time, clocks
determine the offset between themselves and their master Let t represent physical time, given slave device, the offset
o(t) at time t is: o(t) = s(t) – m(t), m(t) is measured time at master at time t, and s(t) is measured time at local clock at time t.
Master periodically broadcasts the current time to other clocks (up to 10 measurements per second in 1588-2008)
CS 414 - Spring 2012
PTP Synchronization
CS 414 - Spring 2012
T1’ – T1 = o + d and T2’ – T2 = -o + d, then we find
o = (T1’ - T1 – T2’ + T2)/2
The clock now knows offset o during the transactions and can correct itself by this agreement with the master clock.
Other Sync Issues Sync must be considered during object acquisition Sync must be considered during retrieval
Sync access to frames of stored video Sync must be considered during transport
If possible use isochronous protocols Sync must be considered at sink
Sync delivery to output devices Sync must consider support of functions such as pause, forward,
rewind with different speeds, direct access, stop or repeat
CS 414 - Spring 2012
Other Sync Issues Sync must be considered during object acquisition Sync must be considered during retrieval
Sync access to frames of stored video Sync must be considered during transport
If possible use isochronous protocols Sync must be considered at sink
Sync delivery to output devices Sync must consider support of functions such as pause, forward,
rewind with different speeds, direct access, stop or repeat
CS 414 - Spring 2012
Reference Models
We need reference models toUnderstand various requirements for multimedia
sync Identify and structure run-time mechanisms to
support execution of sync Identify interface between run-time mechanismsCompare system solutions for multimedia sync
CS 414 - Spring 2012
Synchronization Reference Model Sync model we will be evaluating in detail
is according to Meyer, Effelsberg, Steinmetz:Sync multimedia objects are classified
according to Media level Stream level Object level Specification level
CS 414 - Spring 2012
Media Level (1) Each application operates single continuous
media streams composed of sequence of LDUs Each media stream responsible for intra-stream
synchronization Assumption at this level: device independence Supported operations at this level:
read(devicehandle, LDU)write(devicehandle, LDU)
CS 414 - Spring 2012
Media Level (2) - Examplewindow = open(“videodevice”);movie = open(“file”); while (not EOF (movie) ) {
read(movie, &LDU);if (LDU.time == 20)printf(“Subtitle 1”); else if (LDU.time == 26) printf(“Subtitle2”); write(window, LDU); }
close(window);close(movie);
CS 414 - Spring 2012
Stream Level (1) Operates on continuous media streams
and groups of streams Models inter-stream synchronization for
need of parallel presentation Offers abstractions:
notion of streams, timing parameters concerning QoS for intra-stream
and inter-stream synchronization
CS 414 - Spring 2012
Stream Level (2) Supports operations:
Start(stream), stop(stream), create-group(list-of-streams);
Start(group), stop(group); Setcuepoint(stream/group, at, event);
Classifies implementation according to Support for distribution (end-to-end, local) Support of type of guarantees (best effort, deterministic) Support of types of supported streams (analog, digital)
CS 414 - Spring 2012
Object Level (1) Operates on all types of media and hides
differences between discrete and continuous media Offers abstractions:
Complete sync presentation Computes and executes complete presentation
schedules that include presentation of non-continuous media objects and calls to stream level
Does not handle intra-stream and inter-stream synchronization (relies on media and stream levels)
CS 414 - Spring 2012
Object Level (2) - Example MHEG – Multimedia Hypermedia Experts Group of ISO
Defines representation and encoding of multimedia and hypermedia objects (object-based declarative programming language)
Provides abstractions suited to real-time presentations implemented via multimedia synchronization functionalities
Provides abstracts for real-time exchange implemented with minimal buffering
Evaluates status of objects and performs actions (e.g., prepare, run, stop, destroy)
For time-dependent streams – access to stream level For time-independent streams – direct access the object to present it
Classification of this level according to (a) distribution capabilities, (b) type of presentation schedule, (c) schedule calculation
CS 414 - Spring 2012
Specification Level (1) Open layer included in tools which allow to create sync
specifications Examples:
Synchronization editors, document editors, authoring systems, conversion tools Examples of such tools: multimedia document formatter that produces MHEG
specifications
Classification: Interval-based spec Time-axes based spec Control flow-based spec Event-based spec
CS 414 - Spring 2012
Specification Level (2) - SMIL SMIL – World Wide Web
Consortium’s Synchronized Multimedia Integration Language
SMIL 2.0 and SMIL 3.0 (developed by CWI, PI: D. Bulterman)
Features: Animations, Time Manipulations, Timing and Synchronization, Content Control, Meta-information, Media objects, Linking, Layout, Transitions
http://www.w3.org/TR/SMIL3/
CS 414 - Spring 2012
SMIL
CS 414 - Spring 2012
“Standards, SMIL 2.0, Examples and Comparisons”, D. Bulterman, Siemens Corporate research, Editor: P. Liu, 2002
Summary
CS 414 - Spring 2012
Conclusion Carefully analyze what kind of synchronization
is needed in your multimedia system and application
Determine at which level you need synchronization
Determine what the synchronization requirements should be based on prior experiments
CS 414 - Spring 2012