10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 1

Advanced Multimedia Technology

RoadmapIntroductionChapter 1: Multimedia Network

RTP & RTCPQoS for multimedia network

Chapter 2: Voice and Video Over IP SIP protocolVoIPVideoIP

Chapter 3: MPEG-4 & H264

10/1/2007Nguyen Chan Hung – Hanoi University of

Technology 2

RTP & RTCP

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 3

RTP and related standards

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 4

Types of RTP Sessions

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 5

RTP ExampleConsider sending 64 kbps PCM-encoded voice over RTP. Application collects the encoded data in chunks, e.g., every 20 msec = 160 bytes in a chunk. (= 8000 bytes/sec/50)The audio chunk along with the RTP header form the RTP packet, which is encapsulated into a UDP segment.

RTP header indicates type of audio encoding in each packet;

senders can change encoding during a conference.

RTP header also contains sequence numbers and timestamps.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 6

RTP Implementations

RTP Sender:Uncompressed media data—audio or video— is captured into a buffer, from which compressed frames are produced. Frames may be encoded in several ways depending on the compression algorithm used (e.g. H264, MPEG-4)Compressed frames are loaded into RTP packets for sending.

If frames are large, they may be fragmented into several RTP packets; if frames are small, several frames may be bundled into a single RTP packet. A channel coder may be used to generate error correction packets or to reorder packets before transmission.

After sending the RTP packets, the buffered media of those packets is freed. The sender must buffer data for some time after the corresponding packets have been sent, depending on the codec and error correction scheme used.The sender is responsible for generating periodic status reports for the media streams it is generating, e.g. lip synchronization. It also receives reception quality feedback from other participants and may use that information to adapt its transmission.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 7

RTP Implementations (2)

RTP receiverReceiver is responsible for:

Collecting RTP packets from the network, Correcting any losses, Recovering the timing, Decompressing the media, Presenting the result to the user. Sends reception quality feedback, allowing the sender to adapt the transmission to the receiver, Maintains a database of participants in the session.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 8

Mixers (2)

A mixer has a unique view of the session: It sees all sources as synchronization sources, whereas the other participants see some synchronization sourcesand some contributing sources. In above figure, participant X receives data from three synchronization sources—Y, Z, and M—with A and B contributing sources in the mixed packets coming from M. Participant A sees B and M as synchronization sources with X, Y, and Z contributing to M. The mixer generates RTCP sender and receiver reports separately for each half of the session, and it does not forward them between the two halves. It forwards RTCP source description and BYE packets so that all participants can be identified

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 9

RTP packet format

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 10

RTCP (2)

• For an RTP session there is typically a single multicast address; all RTP and RTCP packets belonging to the session use the multicast address. • RTP and RTCP packets are distinguished from each other through the use of distinct port numbers. • To limit traffic, each participant reduces his RTCP traffic as the number of conference participants increases.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 11

RTCP packet formatFive types of RTCP packets are defined in the RTP specification:

receiver report (RR), sender report (SR), source description (SDES), membership management (BYE), and application-defined (APP). They all follow a common structure: (see figure)

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 12

Receiver ReportThe reception quality feedback in RR packets is useful not only for the sender, but also for other participants and third-party monitoring tools.

The RR feedback allow the sender to adapt its transmissions according to the feedback. Other participants can determine whether problems are local or common to several receivers, Network managers may use monitors that receive only the RTCP packets to evaluate the performance of their networks.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 13

Sender reportFrom the SR, an application can calculate the average payload data rate and the average packet rate over an interval without receiving the data.

The ratio of the two is the average payload size. If it can be assumed that packet loss is independent of packet size, then:

Receiver Throughput = number of packets * average payload size

The timestamps are used to generate a correspondence between media clocks and the NTP

Used for lip-synch

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 14

SDESSource DEScription (SDES) provides participant identification and supplementary details, such as location, e-mail address, and telephone number. The information in SDES packets is typically entered by the user and is often displayed in the graphical user interface of an application Each list of SDES items starts with the SSRC of the source being described, followed by one or more entries with the format shown in Figure.Each entry starts with a type and a length field, then the item text itself in UTF-8 format.The length field indicates how many octets of text are present; the text is not null-terminated.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 15

BYEThe RC field in the common RTCP header indicates the number of SSRC identifiers in the packet. On receiving a BYE packet, an implementation should assume that the listed sources have left the session and ignore any further RTP and RTCP packets from that source. A BYE packet may also contain text indicating the reason for leaving a session, suitable for display in the user interface.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 16

RTCP APP: Application-Defined RTCP Packets

The application-defined packet name is a four-character prefix intended to uniquely identify this extension, with each character being chosen from the ASCII character set. Application-defined packets are used for nonstandard extensions to RTCP, and for experimentation with new features. Experimenters use APP to try new features, and then register new packet types if the features have wider use. Several applications generate APP packets, implementations should be prepared to ignore unrecognized APP packets.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 17

Audio Capture, Digitization, and Framing

Audio capture devices can produce samples with 8-, 16-, or 24-bit resolution, Linear, µ-law or A-law quantization, Rates between 8,000 and 96,000 samples per second, mono or stereo. It may be necessary to convert the media to an alternative format before the media can be used

for example, changing the sample rate or converting from linear to µ-law quantization

Many speech codecs perform voice activity detection with silence suppression

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 18

Video Capture

Most Video codec use inter-frame compression introduce delayYUV to RGB conversion

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 19

Use of Prerecorded Content RTP makes no distinction between live and prerecorded media, and senders generate data packets from compressed frames in the same way First, the sender must generate a new SSRC and choose random initial values for the RTPtimestamp and sequence number. During the streaming process, the sender must be prepared to handle SSRC collisions and should generate and respond to RTCP packets for the stream. Also, if the sender implements a control protocol, such as RTSP, that allows the receiver to pause or seek within the media stream, the sender must keep track of such interactions so that it can insert the correct sequence number and timestamp into RTP data packets

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 20

Fragmentation of a Media Frame into RTP Packets

The fragmentation process is critical to the quality of the media in the presence of packet loss. The ability to decode each fragment independently is desirable

otherwise loss of a single fragment will result in the entire frame being discarded When multiple RTP packets are generated for each frame, the sender must choose between sending the packets in a single burst and spreading their transmission across the framing interval.

Sending the packets in a single burst reduces the end-to-end delay but may overwhelm the limited buffering capacity of the network or receiving host.

it is recommended that the sender spread the packets out in time across the framing interval.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 21

Packet Reception – Input queuesSeparation between the packet reception and playout routines by input queues (See figure)It is important to store the exact arrival time, M, of RTP data packets to calculate interarrival jitter The arrival time should be measured according to a local reference wall clock, T, converted to the media clock rate, R. Since the receiver do not have such a clock, so usually we calculate the arrival time by sampling the reference clock (typically the system wall clock time) and converting it to the local timeline:

where the offset is used to map from the reference clock to the media timeline, in the process correcting for skew between the media clock and the reference clock.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 22

Disruption of Interpacket Timing during Network Transit

There are bursts when several packets arrive at once Gaps when no packets arrive Packets may even arrive out of order.

The receiver does not know when data packets are going to arrive, so it should be prepared to accept packets in bursts, and in any order

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 23

The Playout Buffer

Data packets are extracted from their input queue and inserted into a source-specific playout buffer sorted by their RTP timestamps. Frames are held in the playout buffer for a period of time to smooth timing variations caused by the network. Holding the data in a playout buffer also allows the pieces of fragmented frames to be received and grouped, and it allows any error correction data to arrive. The frames are then decompressed, any remaining errors are concealed, and the media is rendered for the user.A single buffer may be used to compensate for network timing variability and as a decode buffer for the media codec.

It is also possible to separate these functions: using separate buffers for jitter removal and decoding.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 24

The Playout Buffer Data Structures The playout buffer comprises a time-ordered linked list of nodes. Each node represents a frame of media data, with associated timing information. The data structure for each node contains pointers to:

the adjacent nodes, the arrival time, RTP timestamp, desired playout time for the frame,

and pointers to both The compressed fragments of the frame (the data received in RTP packets)

and The uncompressed media data

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 25

Clock skew

Calculation of clock skew: observe the rate of the sender clock—the RTPtimestamp—and compare with the local clock. If TR(n) is the RTP timestamp of the n th packet received, and TL(n) is the value of the local clock at that time, then the clock skew can be estimated as follows:

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 26

The Playout calculation

5 steps:1. The sender timeline is mapped to the local playout timeline, compensating for

the relative offset between sender and receiver clocks, to derive a base time for the playout calculation

2. If necessary, the receiver compensates for clock skew relative to the sender, by adding a skew compensation offset that is periodically adjusted to the base time

3. The playout delay on the local timeline is calculated according to a sender-related component of the playout delay and a jitter-related component

4. The playout delay is adjusted if the route has changed , if packets have been reordered, if the chosen playout delay causes frames to overlap, in response to other changes in the media

5. Finally, the playout delay is added to the base time to derive the actual playouttime for the frame.

10/1/2007 Nguyen Chan Hung – Hanoi University of Technology 27

Key points of Chapter 1

RTP & RTCPMedia transmission and reception over networkTranslator & MixerRTP SessionRTP StreamRTP Packet format

SSRC & CSRCRTCP packet format

SR/RR/SDES/BYE/APP

QoS:Scheduling

WFQPolicing:

Token Bucket Packet Classifications

DSCP/TOSCall Admission

T-Spec/R-SpecInt-Serv

RSVPDiff-Serv

Forwarding PHBAF/EF

DSCPInt-Serv vs. Diff-Serv