multimedia applications: streamingce.sharif.edu/courses/90-91/2/ce873-1/resources... · what is...

Multimedia Applications:

Streaming

Instructor: Hamid R. Rabiee

Spring 2012

Outline

What is Streaming Technology?

Issues in Video Streaming over the Internet

Bandwidth

Loss rate

Delay

Streaming types

Streaming stored multimedia

Streaming live multimedia

Streaming through a web server

Streaming through a streaming server

Real Time Streaming Protocol (RTSP)

2Digital Media Lab - Sharif University of Technology

What is Streaming Technology?

Streaming multimedia allows the user to begin viewing video clips without

completely downloading the entire file.

After a brief initializing and buffering, the file begins to stream.

Streaming process

At Sender: Partition video into packets

At Receiver: After X-second (e.g. 5s to 15s) delay -> begin playback while video is still

being downloaded

At Receiver : Simultaneous delivery and playback (with short delay)

Advantages:

Low delay before viewing

Minimum storage requirements

Disadvantages:

Any data that is lost or arrived late is useless


Issues in Video Streaming over the Internet

Internet only offers best-effort service => Provide no guarantees on

1. Bandwidth

2. Loss rates

3. Delay

1. Bandwidth

Available bandwidth is dynamic

Internet doesn’t provide bandwidth reservation

If transmit faster than available bandwidth

Congestion occurs, packet loss, and severe drop in video quality

If transmit slower than available bandwidth

Sub-optimal video quality

Goal: Match video sending bit rate with available bandwidth = Rate Control

Video bit rate may be adapted by

Varying the quantization

Varying the frame rate

Varying the spatial resolution

Adding/dropping layers (for scalable coding)

Digital Media Lab - Sharif University of Technology4

Issues in Video Streaming over the Internet :

Overcoming Bandwidth Problem Rate control schemes:

Source-based rate control

Source is responsible for adapting the video transmission rate

Probe-based approach: source probes the network for available

bandwidth by adjusting sending rate in a way that

Packet loss ratio p < threshold Pth

If p< Pth then increase transmission rate

If p> Pth then decrease transmission rate

Model-based approach: It is based on the TCP throughput model to determine video

stream’s sending rate


RTT

MTU22.1

Time Trip RoundRTT , ratio lossPacket ρ

Unit Transmit MaximumMTU , TCP of Throughputλ

Fig.1 – An architecture for the

source-based rate control system

Match the rate of pre-compressed

video bitstream to the target rate


Overcoming Bandwidth Problem (cont.) Receiver-based rate control

Receiver is responsible for adapting the video

receiving rate by adding/dropping channels

Useful in multicasting scalable video; each video

layer correspond to one channel in multicast tree

Applying approaches

Probe-based

No congestion detected => receiver probes for available bandwidth => joining a layer => receiving rate

increased

Congestion detected => receiver drops a layer => receiving rate reduced

Model-based

The same as in the sender-based rate control

Hybrid rate control

Receivers regulate the receiving rate of video streams by adding/dropping channels while sender adjusts the

transmission rate of each channel based on the receivers’ feedback

Examples: Destination set grouping, Layered multicast scheme


Source

Client with high-rate connection

Client with medium-rate connection

Client with low-rate connection

Client with low-rate connection

Base layer

Enh layer 1

Enh layer 2

Fig.2 – Example of Receiver-Driven

Layered Multicast


Overcoming Loss Rate Problem2. Packet Loss

Can damage pictures in a video sequence

Goal: Overcome packet loss via error control

Forward Error Correction (FEC)

A video stream chopped into segments, each segment is packetized

into k packet

A block code applied to k packets => n-packet block generated

By receiving any k packets in n-packet block, the segment can be

recovered

Advantages:

Low delay (as compared to retransmits)

Doesn’t require feedback channel

Works well (if appropriately matched to channel)

Disadvantages:

Overhead

Channel loss characteristics are often unknown and time-varying

FEC may be poorly matched to channel. Therefore, it is often ineffective (too little FEC) or inefficient (too much FEC)



Overcoming Loss Rate Problem

3

1 2

41 3

1 2 3 4

1 1 2 2 3 4

12 LOSS

3 4

Original

stream

Redundancy

Received

stream

Reconstructed

stream

Internet


FEC Mechanism

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1 5 9 13 2 6 10 14 3 7 11 15 4 8 12 16

1 5 9 13 2 6 10 14 LOSS 4 8 12 16

1 2 4 5 6 8 9 10 12 13 14 16

Original Stream

Interleaved StreamReceived Stream

Reconstructed Stream



Overcoming Loss Rate Problem (cont.)

Interleaving Mechanism



Retransmission

When receiver detects the loss of packet N:

if ( Tc + RTT + Ds < Td (N)) then “send the request for packet N to sender”

{ Tc = Current time, RTT= Round trip time, Ds= Slack term, Td(N)= time of scheduling packet N for

display }

Two approaches in video streaming with time-sensitive data

Delay-constrained retransmission

Only retransmit packets that can arrive in time

Priority-based retransmission

Retransmit important packets before unimportant packets

Advantages: Only resends lost packets, efficiently uses bandwidth

Easily adapts to changing channel conditions

Disadvantages: Latency (round-trip-time (RTT))

Requires a back-channel (not applicable in broadcast, multicast, or point-to-point w/o back-channel)

For some applications, usefulness decreases with increasing RTT




Error concealment

Performed by receiver when packet loss has already occurred

Approaches to apply error concealment

Spatial interpolation: Missing pixels are reconstructed using neighboring

spatial information

Temporal interpolation: loss data is reconstructed using data in previous

frames

Error-resilient video coding

Using Multiple Description Coding (MDC): a raw video sequence is compressed

into multiple streams

Robust to packet loss: if receiver gets one description, it can reconstruct video with

acceptable quality

Enhanced video quality: if receiver gets multiple description, a better reconstruction can

be made by combining them



Overcoming Delay Problem

3. Delay (Delay Jitter)

Delay jitter:

End-to-end delay may fluctuate from packet to packet

Streaming video needs bounded end-to-end delay so that packets

can arrivein time for decoding and playback

Jitter: Variation in the end-to-end delay

Receiver should decode and display frames at the same rate

Each frame has its own specific playout time

Playout time: Deadline by which it must be

received/decoded/displayed

If video packet arrives late (after its plaback deadline) => it is

useless , can be considered as lost

If subsequent frames depend on the late frame, then effects can

propagate

Goal: Overcome delay jitter => Playback buffer



Overcoming Delay Problem (cont.)


Approach: Add playback buffer at decoder side to compensate for

jitter

Corresponds to adding an offset to the playback time of each packet

If (packet delay < offset) then OK

Buffer packet until its playback time

If (packet delay > offset) then problem

Playback buffer typically has duration of 5-15 secs

Compensates for delay jitter and enables retransmission of lost packets

constant bit

rate video

transmission

time

variable

network

delay (jitter)

client video

receptionconstant bit

rate video

playout at client

client playout

delay

bu

ffe

red

vid

eo




Client-side buffering:



Packet delivery, time-varying delay (jitter), and

playout delay:

15 Digital Media Lab - Sharif University of Technology



Effect of Different Playout Delays

Playout delays: TD1 < TD2 < TD3


Streaming Stored Multimedia

Streaming stored media:

Audio/video file is stored in a

server

Users request audio/video file

on demand.

Audio/video is rendered

within, say, 10 s after request.

Interactivity (pause, re-

positioning, etc.) is allowed.

Media player:

removes jitter

decompresses

error correction

graphical user interface

with controls for

interactivity

Plug-ins may be used to

embed the media player into

the browser window.


Streaming Stored Multimedia (cont.)


18

1. video

recorded

2. video

sent3. video received,

played out at client

streaming: at this time, client

playing out early part of video,

while server still sending later

part of video

network

delaytime

Streaming Live Multimedia

Broadcasting the media, live over the Internet. The process involves

a camera for the media,

an encoder to digitize the content,

a media publisher where the streams are made available to potential end-users

a content delivery network to distribute and deliver the content.

playback buffer is required (as with the streaming stored multimedia)

playback can lag tens of seconds after transmission

still have timing constraint

Interactivity Features

fast forward impossible

rewind, pause possible

Examples:

Internet radio talk show

Live sporting event


Interactive, Real-Time Multimedia

End-end delay requirements: audio: < 150 msec good, < 400 msec OK

includes application-level (packetization) and network delays

higher delays noticeable, impair interactivity

Session initialization how does callee advertise its IP address, port number, encoding algorithms?

Applications: IP telephony, video conference, distributed interactive worlds


Interactive, Real-Time Multimedia (cont.)

Applications:

PC-2-PC phone

instant messaging services are providing this

PC-2-phone

Dialpad

Net2phone

Videoconference with Webcams



Internet Phone, an example of interactive multimedia

speaker’s audio: alternating talk spurts, silent periods.

64 kbps during talk spurt

pkts generated only during talk spurts

20 msec chunks at 8 Kbytes/sec: 160 bytes data

application-layer header added to each chunk.

Chunk+header encapsulated into UDP segment.

application sends UDP segment into socket every 20 msec during

talkspurt.



Internet Phone (cont.)

Network loss: IP datagram lost due to network congestion (router buffer

overflow)

Delay: IP datagram arrives too late for playout at receiver => considered

as lost

delays: processing, queueing in network; end-system (sender, receiver) delays

Typical maximum tolerable delay: 400 ms

loss tolerance: depending on voice encoding, losses concealed, packet loss

rates between 1% and 10% can be tolerated.



packets

time

packets

generated

packets

received

loss

r

p p'

playout schedule

p' - r

playout schedule

p - r

Internet Phone: Fixed Playback Delay

Sender generates packets every 20 msec during talk spurt.

First packet received at time r

First playback schedule: begins at p

Second playback schedule: begins at p’



packetith receivingafter delay network average of estimated

acketpith for delay network tr

receiverat played is ipacket timethep

receiverby received is ipacket timether

packetith theof timestampt

i

ii

i

i

i

)()1( 1 iiii trudud

Adaptive Playout Delay

Goal: minimize playout delay, keeping late loss rate low

Approach: adaptive playout delay adjustment:

Estimate network delay, adjust playout delay at beginning of each talk spurt.

Silent periods compressed and elongated.

Chunks still played out every 20 msec during talk spurt.

Dynamic estimate of average delay at receiver (u is a fixed constant ):



||)1( 1 iiiii dtruvuv

iiii Kvdtp

Adaptive Playout Delay (cont.)

Also useful to estimate the average deviation of the delay, vi :

The estimates di and vi are calculated for every received packet,

although they are only used at the beginning of a talk spurt.

For first packet in talk spurt, playout time is:

where K is a positive constant.

Remaining packets in talkspurt are played out periodically


Streaming from a Web server

Audio and video files stored in Web servers

naive approach

browser requests file with HTTP request

message

Web server sends file in HTTP response

message

content-type header line indicates an

audio/video encoding

browser launches media player, and passes file

to media player

media player renders file

Major drawback: media player

interacts with server through

intermediary of a Web browser


Streaming from a Web server (cont.)

Alternative: set up connection between

server and player

Web browser requests and receives a

meta file

(a file describing the object) instead

of receiving the file itself;

Content-type header indicates

specific audio/video application

Browser launches media player and

passes it the meta file

Player sets up a TCP connection with

server and sends HTTP request.

Some concerns:

Media player communicates over

HTTP, which is not designed with

pause, ff, rwnd commands

May want to stream over UDP

These concerns may not be real

problems – more later


Streaming from a streaming server

This architecture allows for

non-HTTP protocol between

server and media player

Can also use UDP instead of

TCP.


Streaming Multimedia: UDP or TCP?

UDP

server sends at rate appropriate for client (oblivious to network congestion !)

often send rate = encoding rate = constant rate

then, fill rate = constant rate - packet loss

short playback delay (2-5 seconds) to compensate for network delay jitter

error recover: time permitting

TCP

send at maximum possible rate under TCP

fill rate fluctuates due to TCP congestion control

larger playback delay: smooth TCP delivery rate

HTTP/TCP passes more easily through firewalls


Real Time Streaming Protocol (RTSP)

A signaling and control protocol for multimedia streaming in Internet

To control the data delivery in a multimedia streaming session by conveying

VCR-style commands (like play, mute) between communicating partners

It is typically used in conjunction with RTP which conveys the actual

multimedia data.

It is a request-response protocol similar to HTTP, but stateless

Server and player use RTSP to send control info to each other

For user to control display: rewind, fast forward, pause, resume, repositioning, etc…

What it doesn’t do:

Does not define how audio/video is encapsulated for streaming over

network

Does not restrict how streamed media is transported; it can be

transported over UDP or TCP

Does not specify how the media player buffers audio/video


RTSP Request

Has the form:

Request-Method SP Request-URL SP RTSP-Version <CR><LF>

(generic-header | request-header | entity-header <CR><LF>)

<CR><LF>

[message body]

Request-Method is:

DESCRIBE - retrieves the description of a media object from a

server;

SETUP - prepares the streaming session;

PLAY - starts the delivery of multimedia data;

PAUSE - streaming is paused, session is still active, but no packet is

sent;

TEARDOWN - session is terminated and resources are freed.


RTSP Request (cont.)

Request-header can have the following fields (selection):

Accept : MIME types of resources accepted by client

Accept-Encoding : encoding accepted by client

Accept-Language : language accepted by client

Authorization : user-agent wishes to authenticate itself with a

server

From:

Referer : the URL of document refering this URL

User-Agent : client software


RTSP Response

Has the form:

Http-Version SP Status-Code SP Reason-Phrase<CR><LF>

(generic-header | response-header | entity-header <CR><LF>)

<CR><LF>

[message body]

Response-header has the following fields (selection):

Location : redirect the client to a location other than Request-URL

for completion of the request

Retry-After : indicate to client how long the service is expected to be

unavailable

Server : information about software used by the server to handle the

request


RTSP: out of band control

FTP uses an “out-of-band” control

channel:

A file is transferred over one channel.

Control information (directory changes, file

deletion, file renaming, etc.) is sent over a

separate TCP connection.

The “out-of-band” and “in-band” channels

use different port numbers.

RTSP messages are also sent out-of-band:

The RTSP control messages use different

port numbers than the media stream, and

are therefore sent out-of-band.

The media stream, whose packet structure

is not defined by RTSP, is considered “in-

band”.

If the RTSP messages were to use the same port

numbers as the media stream, then RTSP

messages would be said to be “interleaved” with

the media stream.


RTSP initiates and controls delivery

Client obtains a description of the multimedia

presentation, which can consist of several media streams.

The browser invokes media player (helper application)

based on the content type of the presentation description.

Presentation description includes references to media

streams, using the URL method rtsp://

Player sends RTSP SETUP request; server sends RTSP

SETUP response.

Player sends RTSP PLAY request; server sends RTSP

PLAY response.

Media server pumps media stream.

Player sends RTSP PAUSE request; server sends RTSP

PAUSE response.

Player sends RTSP TEARDOWN request; server sends

RTSP TEARDOWN response.

HTTP GET

SETUP

PLAY

media stream

PAUSE

TEARDOWN

media

player

Web

server

media

server

Web

browser

client server

presentation desc.

Meta file example

<title>Twister</title>

<session>

<group language=en lipsync>

<switch>

<track type=audio

e="PCMU/8000/1"

src = "rtsp://audio.example.com/twister/audio.en/lofi">

<track type=audio

e="DVI4/16000/2" pt="90 DVI4/8000/1"

src="rtsp://audio.example.com/twister/audio.en/hifi">

</switch>

<track type="video/jpeg"

src="rtsp://video.example.com/twister/video">

</group>

</session>


RTSP session

Each RTSP has a session identifier,

which is chosen by the server.

The client initiates the session with

the SETUP request, and the server

responds to the request with an

identifier.

The client repeats the session

identifier for each request, until the

client closes the session with the

TEARDOWN request.

RTSP port number is 554.

RTSP can be sent over UDP or

TCP. Each RTSP message can be

sent over a separate TCP

connection.


RTSP: exchange example

C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0

Transport: rtp/udp; compression; port=3056; mode=PLAY

S: RTSP/1.0 200 1 OK

Session 4231

C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0

Session: 4231

Range: npt=0-

C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0

Session: 4231

Range: npt=37

C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0

Session: 4231

S: 200 3 OK

npt = Normal play time


Next Session

Multimedia Protocols


multimedia applications: streamingce.sharif.edu/courses/90-91/2/ce873-1/resources... · what is...

Documents