school of computing science simon fraser university, canada
DESCRIPTION
School of Computing Science Simon Fraser University, Canada. Rate-Distortion Optimized Streaming of Fine-Grained Scalable Video Sequences Mohamed Hefeeda & ChengHsin Hsu MMCN 2007 31 January 2007. Motivations. Multimedia streaming over the Internet is becoming very popular - PowerPoint PPT PresentationTRANSCRIPT
February 2000 1
School of Computing ScienceSchool of Computing ScienceSimon Fraser University, CanadaSimon Fraser University, Canada
Rate-Distortion Optimized Streaming of Fine-Grained Scalable Video Sequences
Mohamed Hefeeda & ChengHsin Hsu
MMCN 2007
31 January 2007
2
MotivationsMotivations
Multimedia streaming over the Internet is becoming very popular
- More multimedia content is continually created- Users have higher network bandwidth and more
powerful computers
Users request more multimedia content And they look for the best quality that their
resources can support
3
Motivations (cont’d)Motivations (cont’d)
Users have quite heterogeneous resources (bandwidth)
- Dialup, DSL, cable, wireless, …, high-speed LANs To accommodate heterogeneity scalable video
coding: Layered coded stream
- Few accumulative layers- Partial layers are not decodable
Fine-Grained Scalable (FGS) coded stream- Stream can be truncated at bit level
4
Motivations (cont’d)Motivations (cont’d)
Goal: Optimize quality for heterogeneous receivers
In general setting- FGS-coded streams - Multiple senders with heterogeneous bandwidth and
store different portions of the stream
Why multiple senders?- Required in P2P streaming:
• Limited peer capacity and Peer unreliability
- Desired in distributed streaming environment:• Disjoint network path Better streaming quality
5
Our Optimization ProblemOur Optimization Problem
Assign to each sender a rate and bit range to transmit such that the best quality is achieved at the receiver.
Consider a simple example to illustrate the importance of this problem
6
Example: Different Streaming SchemesExample: Different Streaming Schemes
Non-scalable Layered
7
Example: Different Streaming SchemesExample: Different Streaming Schemes
Optimal FGS ScalableFGS Scalable
1. FGS enables us to get the best quality from senders
2. However, there too many allocation options, and we need to carefully choose the optimal one
8
Problem FormulationProblem Formulation
First: single-frame case- Optimize quality for individual frames
Then: multiple-frame case- Optimize quality for a block of frames- More room for optimization
- Details are presented in the extended version of the paper
9
Input ParametersInput Parameters
T : fixed frame period n : number of senders bi : outgoing bandwidth of sender i
bI : incoming bandwidth of receiver
si : length of (contiguous) bits held by sender i
Assume w.l.g. s1 <= s2 <= …… <= sn
10
Allocation: A = {(Δi , ri) | i = 1, 2, …, n}- Δi : number of bits assigned to i
- ri : streaming rate assigned to i
Specifies:- Sender 1 sends range [0, Δ1 -1] at rate r1
- Sender 2 sends range [Δ1 , Δ1+Δ2 -1] at rate r2
- …- Sender i sends range at rate ri
OutputsOutputs
]1,[1
1
1
−ΔΔ ∑∑=
−
=
i
tt
i
tt
11
Integer Programming ProblemInteger Programming Problem
Minimize distortion Subject to:
- on-time delivery- assigned range is
available- assigned rate is
feasible- Aggregate rate not
exceeds receiver’s incoming BW
minA
D( Δt)t=1
n∑s.t. Δi −riT ≤0
Δtt=1
i∑ ≤si ri ≤bi
rtt=1
n∑ ≤bI
Δi,ri ∈N; i =1,2,L ,n.
12
How do we Compute Distortion?How do we Compute Distortion?
Using Rate-Distortion (R-D) models- Map bit rates to perceived quality- Optimize quality rather than number of bits
Approaches to construct R-D models- Empirical Models: Many empirical samples expensive
- Analytic Models: Quality is a non-linear function of bit rate, e.g., log model [Dai 06] and GGF model [Sun 05]
- Semi-analytic Models: A few carefully chosen samples, then interpolate, e.g., piecewise linear R-D model [Zhang 03]
Detailed analysis of R-D models in our previous work [Hsu 06]
13
Within each bitplane, approximate R-D function by a line segment Line segments of different bitplanes have different slopes
The Linear R-D ModelThe Linear R-D Model
14
Visual Validation of Linear R-D ModelVisual Validation of Linear R-D Model
Mother & Daughter, frame 110 Foreman, frame 100
15
Rigorous Validation of Linear R-D ModelRigorous Validation of Linear R-D Model
Average error is less than 2% in most cases
16
Let yi be number of bits transmitted from bitplane i
Distortion is:
- d : base layer only distortion- gi : slope of bitplane i
- z : total number of bitplanes
Using the Linear R-D ModelUsing the Linear R-D Model
d + ghyhh=1
z∑
17
Integer Linear Programming (ILP) ProblemInteger Linear Programming (ILP) Problem
Linear objective function
Additional constraints- number of bits
transmitted from bit plane h does not exceed its size lh
- bits assigned to senders are divided among bitplanes
minA
D( Δt)t=1
n∑ =d + gvyvv=1
z∑s.t. Δi −riT ≤0
Δtt=1
i∑ ≤si ri ≤bi
rtt=1
n∑ ≤bI
yh ≤lh
Δtt=1
n∑ ≤ yvv=1
z∑Δi,ri, yh ∈N; i =1,2,L ,n; h=1,2,L ,z.
18
Solution of ILP is a Valid FGS StreamSolution of ILP is a Valid FGS Stream
Lemma 1: - An optimal solution for the integer linear program produces a
contiguous FGS-encoded bit stream with no bit gaps
Proof sketch- minimizing
- Since g1 < g2 < … <gn<0 (line segment slopes),
- the ILP will never assign bits to yi+1 if yi is not full
d + ghyhh=1
z∑
19
Solving ILP problem is expensive Solution: Transform it to Linear Programming (LP)
problem - Relax variables to take on real values
Objective function and constraints remain the same
Linear Programming RelaxationLinear Programming Relaxation
Δ i ,ri , yh ∈R+ U{0}; i = 1,2,L ,n; h = 1,2,L , z.
20
Solve LP - Result is real values
Then, use the following rounding scheme for solution of the ILP
Efficient Rounding SchemeEfficient Rounding Scheme
A ={(Δi, ri)|i =1,2,L ,n}
ri = ri⎢⎣ ⎥⎦,
Δi =Δi
ri −1ri
⎢⎣⎢
⎥⎦⎥, r >1
0, 0 ≤r≤1
⎧⎨⎪
⎩⎪
21
Correctness/Efficiency of Proposed RoundingCorrectness/Efficiency of Proposed Rounding
Lemma 2 (Correctness) - Rounding of the optimal solution of the relaxed
problem produces a feasible solution for the original problem
Lemma 3 (Efficiency: Size of Rounding Gap) - The rounding gap is at most nT + n, where n is the
number of senders and T is the frame period
- (Extreme) Example: n = 30 senders, T = 30 fps ==> gap is 32 bits
- Indeed negligible (frame sizes are in order of KBs)
22
FGSAssign: Optimal Allocation AlgorithmFGSAssign: Optimal Allocation Algorithm
Solving LP (using Simplex method for example) may still be too much
- Need to run in real-time on PCs (not servers)
Our solution: FGSAssign- Simple, yet optimal, allocation algorithm- Greedy: Iteratively allocate bits to sender with
smallest si (stored segment) first
23
Pseudo Code of FGSAssignPseudo Code of FGSAssign
1. Sort senders based on si, s1 ≤ s2 ≤ …… ≤ sn;2. x0 = …… = xn = 0; Δ1 = …… = Δn = 0; ragg = 0;3. for i = 1 to n do4. xi = min(xi−1 + biT, si);5. ri = (xi − xi−1)/T ;6. if (ragg + ri < bI ) then7. ragg = ragg + ri;8. Δi = xi − xi−1;9. else10. ri = bI − ragg;11. Δi = T × ri;12. return13. endfor
24
Optimality of FGSAssignOptimality of FGSAssign
Theorem 1- The FGSAssign algorithm produces an optimal
solution in O(n log n) steps, where n is the number of senders.
Proof: see paper
Experimentally validated as well.
25
Multiple-Frame OptimizationMultiple-Frame Optimization
Solve the allocation problem for blocks of m frames each
Objective: minimize total distortion in block
Why consider multiple-frame optimization?- More room for optimization- Solve the problem less often
26
Multiple-Frame Optimization: Why?Multiple-Frame Optimization: Why?
More room for optimization: higher quality and less quality fluctuation
Bit rate
Dis
torti
on
(MS
E)
27
Multiple-Frame OptimizationMultiple-Frame Optimization
Formulation (in extended version of the paper): - Straightforward extension to single-frame with lager number
of variables and constraints- Computationally expensive to solve
Our Solution: mFGSAssign algorithm - Heuristic (close to optimal results) - Achieves two goals:
• Minimize total distortion in a block of frames• Reduce quality fluctuations among successive frames
- Pseudo code and analysis: see extended version of the paper
28
Experimental SetupExperimental Setup
Software used - MPEG-4 Reference Software ver 2.5
• Augmented to extract R-D model parameters
Algorithms implemented (in Matlab)- LP solutions using Simplex for the single-frame and
multiple-frame problems- FGSAssign algorithm- mFGSAssign algorithm- Nonscalable algorithm for baseline comparisons
29
Experimental Setup (cont’d)Experimental Setup (cont’d)
Streaming scenarios- Four typical scenarios for Internet and corporate
environments
Testing video sequences - Akiyo, Mother, Foreman, Mobile (CIF)- Sample results shown for Foreman and Mobile
30
Single Frame: Quality (PSNR)Single Frame: Quality (PSNR)
Foreman, Scenario I Mobile, Scenario III
Quality Improvement: 1--8 dB FGSAssign is optimal
31
Multiple Frame: Quality (PSNR)Multiple Frame: Quality (PSNR)
Foreman, Scenario II Mobile, Scenario III
Scalable: higher improvement than single frame mFGSAssign: almost optimal (< 1% gap)
32
Fluctuation ReductionFluctuation Reduction
Foreman, Scenario II Mobile, Scenario III
Small quality fluctuations in successive frames
33
ConclusionsConclusions
Formulated and solved the bit allocation problem to optimize quality for receivers in general settings
- Multiple heterogeneous senders
Considered single and multiple frame cases Approach
- Nonlinear problem integer linear program• Using linear R-D model
- Integer linear program linear program • Using simple rounding scheme
Proposed efficient algorithms - FGSAssign: optimal and efficient- mFGSAssign: close to optimal in terms of average distortion, reduces
quality fluctuations, runs in real time
Significant quality improvements shown by our experiments
34
Thank You!Thank You!
Questions??
All programs/scripts/videos are available:
http://www.cs.sfu.ca/~mhefeeda