rate-control in video codec heejune ahn embedded communications laboratory seoul national univ. of...
TRANSCRIPT
Rate-Control in Video Codec
Heejune AHNEmbedded Communications Laboratory
Seoul National Univ. of TechnologyFall 2008
Last updated 2008. 11. 16
Heejune AHN: Image and Video Compression p. 2
Agenda
Basic Concepts Rate-Distortion Theory Practical Rate-Control Algorithms Other issues
Heejune AHN: Image and Video Compression p. 3
1. Rate Control Concepts
Key players coded video quality: (usu. Lower limits) bit-rate: some constraints (usu. upper limits) input video sequence: (noisy, degree of details, motion complexity) coding parameters (standards, coded mode, quantization step)
Rate-optimization select the parameters for best video quality under constraints Poor algorithm : low quality, large fluctuation, dropped frames
encoder
Input Video Coded Video
Bit-rate contraints
network/
storage
Rate Control
Coding para. selection
Heejune AHN: Image and Video Compression p. 4
Video stream Bit-rate characteristics
Factors Encoding algorithms
(intra/inter, forward/bi-direction, DCT/Wavelet , standards, etc)
Input video Encoding parameters
• Q step, picture &mb mode, mv search area, GOP structures
4000 4200 4400 4600 4800 50001.0
1.5
2.0
2.5
4000 4200 4400 4600 4800 50000
1
2
3
0
1
2
3
화면 번호
화
면당
트래
픽 (
x 10
5 bi
ts/f
ram
e)
(c) MPEG-1 기법으로 부호화
(b) H.261 기법으로 부호화
(a) JPEG 기법으로 부호화
4000 4200 4400 4600 4800 5000
Heejune AHN: Image and Video Compression p. 5
Bit-rate Constraints
Typical constraints parameters Average (mean) bit-rate (with duration) Peak (maximum) bit-rate Maximum variation in bit rate Prevention of underflow and overflows at encoder &decoder buffer Required latency
Examples DVD video
• Total 4.7GB (play time 3 hours), peak rate => 3.5Mbps with some peak limits
Video conferencing in ISDN or Circuit switching network• Constant transmission bit-rate and delay => Based on buffer size, peak, average rate is calcuated
Vide streaming over internet and packet switched network• Network condition is variable. Should be adaptive rate control
Heejune AHN: Image and Video Compression p. 6
2. Rate Control Theory
Revisit to Quantization For Uniform distributed Input & Uniform quantizer
Rate Distortion Theory General trends are same as above
2 , 2 )2(3
12
12
1
1222
2
222R
xR
rN
A
N
AD
distortion ~ input signal variance and exp. desc.(bitrate) !!
Heejune AHN: Image and Video Compression p. 7
Rate-Distortion Curve Rate decreases as distortion increases
• depends on input statistics (complex image => large variance)
• PSNR measure is 10log10(2552/D)
3.0
2.0
1.0
0.1 0.2 0.4 1
R DI( ) ( )
D / 2
0.1 0.5 1.0
)()( RD I
R
2
R D DD
D
I( ) ( )log ,
,
1
20
0
22
2
Heejune AHN: Image and Video Compression p. 8
Rate-Distortion Optimization
Motivation Which subject I have to study if I gets 90pts in Math, 50pts in Eng. ? Image coding
• Condition: we can use 10KBytes for one picture and each parts have different complexity (i.e, simpler and complex, variance)
• Which parts I have to assign more bits?
simpler blocks
More complex blocks
Heejune AHN: Image and Video Compression p. 9
Lagrange multiplier
Constrained Optimization Problem Minimize distortion with bits no larger than max bits
Cannot use partial differential for minima and maxima Lagrange Multiplier technique
Insert one more imaginary variable (called Lagrange) We have multivariable minimization problem
maxR R subject to )}(min{ RD
})(min{ RRDJ
Heejune AHN: Image and Video Compression p. 10
Side note: why not partial differential
Simple example min subject to
Solution• By substitution• minimum ½ at x = y = 1/sqrt(2)
Wrong Solution With complex contraints, we cannot use substitution of variable
method We cannot change partial different with ordinary difference
• If we check,
• x = y = 0 ?
22 yx 1
122),( yxyxf 1),( yxyx
02)( 22
xx
yx 02)( 22
yy
yx
Heejune AHN: Image and Video Compression p. 11
An Example: Lagrange Multiplier
Optimization Problem multiple independent Gaussian variables
Lagrange multiplier optimization
BbM
ii
)(1
i hλb
J
B b b DJ
ibi
i
M
ii
ii
22
1
2)2ln( 2 0
)()(
1 ,22)( ),()( 2222
1
hhhbDbDBD ii b
ib
iiii
M
iii
iChibi for ])2 ln 2[( 2 122
Heejune AHN: Image and Video Compression p. 12
Solution M 개를 모두 곱하고 M square
를 하면
Bit-allocation
Lagrange value
CbM
bMbM
ii
i
ii
222
2
12
1
2 222
M
bb i
M
ii
,
1
22
2
2
2 log2
1
0 i
i bb
bM
ii
b hh 2
1
222 22ln222ln2
Heejune AHN: Image and Video Compression p. 13
RDO in Image Coding
Bit allocation problem Modified Lagrange Method First proposed by H. Everett, then signal processing domain by
Shoham and Gersho Theorem
For any the solution B*(to the ‘unconstrained’ problem
is also a solution to the ‘constrained’ problem
subject to
Here B = {r1,r2,. . ., rN}, ie. Bit allocaton
each ri such that
])},,,([]),,,([{min 2121 NNSB rrrRrrrD
)}({min RDSB
CRBR )(
*)( ii
rDr
Heejune AHN: Image and Video Compression p. 14
RDO algorithm
How to find such that results to Algorithm
Set Get Ri for each i independently ie. tagent to slope of RD curve Get R (e.g. sum of ri’s) Repeat until R = Rc, changing (note: large small b)
Diverse Constraints Sum Ri < R Embedded sum Ri < Rx Buffer controls Dependency condition
*)( ii
rDr
CRBR *)(
Heejune AHN: Image and Video Compression p. 15
RDO algorithm
Intuitive understanding Reverse Water filling theory
• larger bits for larger variance block
• The returns will be same at the equivalent point
Large variance
RD curve
small variance
RD curve
RS* RL*
Heejune AHN: Image and Video Compression p. 16
Practical Rate Control
Problem in Lagrange optimization too much computational complexity for real-time use
Rate control Model Buffer occupancy level B at the end of i-th frame
• BiPRE = Bi-1 + Rv – Rc
• Bi = BiPRE if 0< Bi
PRE <Bmax
= Bmax if overflow (BiPRE >Bmax)
= 0 if underflow (BiPRE < 0)
Rate Control (a negative feedback loop) : see Fig. 10.9• High level => large Q step => smaller Rv => level down• Low level => smaller Q step => larger Rv => level up
Rate Control
Encoder
Quant step
Rv Rc
Input Video
Heejune AHN: Image and Video Compression p. 17
Practical Rate Control
H263 Test Model Rate control Frame Level
• Bit allocation for preventing buffer underflow and overflow (some times even frame skip)
Macroblock level • Allocate bits for each MB
• A : # of pixles in a MB, : standard deviation, Qi : quant. step
• Step 1: calculate the MB complexity
• Step 2: calculate Qi from K, A, C, a,
• Step 3: encode MB• Step 4: update the model parameters K, C
)(2
2
CQ
KABi
ii
Heejune AHN: Image and Video Compression p. 18
MPEG-2 Test Model 5 Bit allocation
• Assign a target bits to current GOP
• Assign to each current picture– Based on Complexity of previous I, P, B pictures (Q-Step and
resultant bit)
Rate control • Compare coded bit with expected values, and adjust Q-Step
Modulation • block activity, increase Q-step
bbbpppiii QSXQSXQSX ,,
PR
C
KX
XKNN
BT
PR
C
KX
XKNN
BT
PR
C
KX
XN
KX
XNB
T
pb
pbpb
rb
bp
bpbp
rp
bi
bb
pi
pp
ri 8
,max,8
,max,8
,1
max
cntMB
jTBdd
cntMB
jTBdd
cntMB
jTBdd b
jbjbp
jpjpi
jiji _
)1(,
_
)1(,
_
)1(101010
r
dQ j
j
31
actavgact
actavgactactN
j
jj _2
_2_
Heejune AHN: Image and Video Compression p. 19
4. Other Rate Control Issue
Variable Complexity Algorithm Constraints on computational power
• Motion search algorithm & search range • Rate Control Mode selection • Frame rate (skipping)
Trade off• Computation complexity v.s. video quality • Also depends on input image (time-varying)
Usage • Computation limited Mobile device (usu. Software based codec)• Iain E.G. Richardson, the athor of textbook,’s interest
Performance Measure PSNR is not enough (for subject quality measure) Some objective measures are under development SSIM(Structural SIMiliarity)
Appendix Rate distortion theory
Heejune AHN: Image and Video Compression p. 21
Rate-Distortion Theory
Distortion of X’ from X D = H(X) – H(X|X’)
• X 전체의 정보양에서 X’ 을 통하여 확보한 X 의 정보양을 뺀것 (make sense? Yeh ~~~)
• 참고 : mutual information : I (X:X’) = H(X:X’) = H(X’:X)
Information theoretic definition
I(X:X’)
H(X’)H(X)
D = H(X) – I(X:X’)
Note: 붉음색은 X’ = Quant(X) 인경우
Heejune AHN: Image and Video Compression p. 22
Rate Distortion
Rate-distortion function Inverse relation between Information/Data Rate and Distortion Example with Gaussian R.V. (known as most difficult distribution)
• Refer to element of Information theory
3.0
2.0
1.0
0.1 0.2 0.4 1
R DI( ) ( )
D / 20.1 0.5 1.0
)()( RD I
R
2
R D DD
D
I( ) ( )log ,
,
1
20
0
22
2
Heejune AHN: Image and Video Compression p. 23
Derivation