lecture 6 scalar and vector quantizationmapl.nctu.edu.tw/course/vc_2009/files/06.lecture 6...lecture...
TRANSCRIPT
Lecture 6 Scalar and Vector Quantization
Wen-Hsiao Peng, Ph.D
Multimedia Architecture and Processing Laboratory (MAPL)Department of Computer Science, National Chiao Tung University
May 2008
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 1 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantization
Lossy Compression Method
Reduce distinct output values to a much smaller setMap an input value/vector to an approximated value/vector
Approaches
Scalar Quant. - quantize each sample separately
Uniform vs. Non-uniformMSE vs. MAE vs. ....
Vector Quant. - quantize a group of samples jointly
Uniform (Lattice Quantizer) vs. Non-uniformMSE vs. MAE vs. ....
Design Objectives
1 Minimize Distortion s.t. Fixed Number of Reconstructions2 Minimize Entropy Rate s.t. Distortion Constraint (Minimize Distortions.t. Entropy Rate Constraint)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 2 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Example
Scalar Quantization
Quantization Levels L = 2R
Reconstruction Values gl , l = 1, 2, ..., LBoundary Values bl , l = 0, 1, ..., LQuantizer Mapping Function
Q(f ) = gl if f 2 Bl = [bl�1, bl )
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 3 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantizer Mapping Function
Q(f ) = gl if f 2 Bl Q(f ) =jf�fminq
k�q+ q
2+f min
Non-uniform Quantizer Uniform Quantizer
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 4 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Uniform vs. Non-uniform
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 5 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Midright vs. Midtread Quantizers
Midright Uniform Quantizer (Even L)
"0" is a decision boundary
Midtread Uniform Quantizer (Odd L)
"0" is a reconstruction levelBetter for small L in video/image coding
Midright vs. Midtread (for Symmetric Distributions)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 6 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantizer Distortion
Dq = σ2q(granular )| {z }Our Focus
+ σ2q(overload)| {z }0 for bounded inputs
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 7 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Quantizer Distortion
Quantizer Distortion Dq1
Dq = E (d(F ,Q(F )))
= ∑l2L
�Zf 2Bl
d(f ,Q(f ))p(f )df
�= ∑
l2LP(Bl )
Zf 2Bl
d(f ,Q(f ))p(f jf 2 Bl )df| {z }Dq,l
Example: d(F ,Q(F )) = (F �Q(F ))2
Dq = E (d(F ,Q(F )) = E�(F �Q(F ))2
�| {z }
MSE
= σ2q
where Quant. Error F �Q(F ) is of zero mean1Depend on distortion measure d(�) and source distribution p(f )
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 8 / 61
Lecture 6 Scalar and Vector Quantization Introduction
Scalar Quantizer
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 9 / 61
Lecture 6 Scalar and Vector Quantization Uniform Scalar Quantizer
Uniform Scalar Quantizer
Equal Quantization Stepsize
bl � bl�1 = gl � gl�1 = q
where
bl = fmin + l � q
gl =bl + bl�1
2Q(f ) =
�f � fminq
��q+q
2+f min| {z }
Closed�form
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 10 / 61
Lecture 6 Scalar and Vector Quantization Uniform Scalar Quantizer
Uniform Quantizer Optimized for Uniform Distribution
Uniform Distribution
p(f ) =
�1/B f 2 (fmin, fmax)0 otherwise
where B = fmax � fmin
Distortion Measure
d(F ,Q(F )) � (F �Q(F ))2
Quantizer Distortion
Dq = ∑l2LP(Bl )Dq,l = Dq,l =
q2
12= σ2f 2
�2R
SNR = 10 logσ2fDq
= (20 log 2)R = 6.02R
where
Dq,l =q2
12, q =
B
L, L = 2R , Signal Variance σ2f = B
2/12
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 11 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
Minimum Mean Square Error Scalar Quantizer
Distortion Measure
d(F ,Q(F )) � (F �Q(F ))2
Objective
(b�, g�) = arg minfb,gg
∑l2L
Z bl
bl�1(f � gl )2p(f )df
!| {z }
J(b,g)
whereb = (b0, b1, ..., bL)
T , g = (g1, g2, ..., gL)T
Necessary Conditions
rJ(b�, g�) = 0) (1) ∂J(b�, g�)/∂bl = 0(2) ∂J(b�, g�)/∂gl = 0
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 12 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
MMSE Scalar Quantizer
Solution
(1) ∂J(b�, g�)/∂bl = (b�l � g �l )2 p(b�l )� (b�l � g �l+1)2p(b�l ) = 0
(2) ∂J(b�, g�)/∂gl = �Z b�l
b�l�1
2(f � g �l )p(f )df = 0
Nearest-Neighbor Condition
b�l =g �l + g
�l+1
2) Bl = ff : d(f , gl ) � d(f , gl 0), 8l 0 6= lg (1)
Centroid Condition
g �l =1
P(Bl )
Z b�l
b�l�1
f p(f )df = E (FjF 2 Bl )| {z }Conditional Mean
(2)
Eqs. (1) & (2) generally DO NOT have a closed-form solution
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 13 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
Properties of MMSE Scalar Quantizer
Symbols
F : original random variableG: quantized random variableQ = F � G: quantization error
G is unbiased estimator of FE (G) = E (F ),E (Q) = 0
G is orthogonal to Q, whereas F is correlated to QE (GQ) = 0,E (FQ) 6= 0
Reduced signal variance
σ2G = σ2F � σ2Q
Equalized error contribution
P(Bl )Dq,l =DqL, 8l 2 L
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 14 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer
MMSE Scalar Quantizer of Various Sources
All sources are with zero mean and unit variance2
Quantizer Design (y j= g j , xj = bj ) SNR
U: Uniform, G: Gaussian, L: Laplacian, Γ: Gamma2See Appendix for arbitrary PDFWen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 15 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer High Rate Approximation
High Rate Approximation: p(f ) is approximately at in all intervals Bl
Dq = ∑l2L
Z bl
bl�1(f � gl )2 p(f )|{z} df
' ∑l2Lp(gl )| {z }
Z bl
bl�1(f � gl )2df
= ∑l2L
P(Bl )∆l| {z }
Z bl
bl�1(f � gl )2df , where ∆l = bl � bl�1
Centroid Condition
∂Dq∂gl
= 0)Z bl
bl�1(f � gl )df = 0) g �l =
bl + bl�12| {z }
High Rate
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 16 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Quantizer Distortion High Rate Approximation
Quantizer Distortion
Dq ' ∑l2L
P(Bl )∆l
Z bl
bl�1(f � g �l )2df = ∑
l2LP(Bl )
∆2l12|{z}Dq,l
where ∆l = bl � bl�1Equalized Error Contribution
P(Bl )Dq,l =1
12P(Bl )"
∆2l#= constant
Special Case: Uniform Quantizer and Uniform Source
P(Bl ) = P(Bl 0),∆l = ∆l 0 = ∆,Dq =∆2
12
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 17 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Quantizer Distortion High Rate Approximation
Alternative Expression (More Useful)
Dq ' ∑l2LP(Bl )
∆2l12' ∑
l2L
pF (g�l )∆
3l
12' σ2f 2
�2Rε2 (See Appendix)
where
ε2 =1
12
�Z fmax/σf
fmin/σf
3
qpF 0(f )df
�3pF 0(f ) = σf pF (σf f ) is PDF of an unit variance source
Uniform Quantizer/Source vs. Non-uniform Quantizer/Source
Uniform : SNR ' 10 log σ2fDq
= 10 log 22R = 6.02R
Non-uniform : SNR ' 10 log σ2fDq
= 6.02R � 20 log ε
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 18 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer of Various Sources
∆SNR = 6.02R (Uniform Source and Quantizer)�maxfSNRgSolid (Non-uniform Quantizer) vs. Dashed (Uniform Quantizer)
Uniform quantizer becomes increasingly ine�cient with increasing R
Non-uniform quantizer attains an asymptote with increasing R
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 19 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
Companding: Compressing and Expanding
A technique for analyzing
quantizers at high rate
Companding Law c(x) de�nes Bl a
Interval Intensity
dc(gl )
dx� 2xmax
L∆l
where
c(xmax) = xmax,c(0) = 0,c(xmin) = xmin
aQuantizers designed by c(x) is nolonger common
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 20 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
Companding: Compressing and Expanding
Distortion Measure
d(F ,Q(F )) � (F �Q(F ))2
Quantizer Distortion (in terms of c(x))
Dq ' ∑l2LP(Bl )
∆2l12
' x2max3L2 ∑
l2Lp(gl )∆l
�dc(gl )
dx
��2' x2max
3L2
Z xmax
xminp(x)
�dc(x)
dx
��2dx
Constraint Z xmax
xmin
dc(x)
dxdx = c(xmax)� c(xmin) = 2xmax
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 21 / 61
Lecture 6 Scalar and Vector Quantization MMSE Scalar Quantizer High Rate Approximation
MMSE Scalar Quantizer Design Using Companding
MMSE Scalar Quantizer
minx2max3L2
Z xmax
xminp(x)
�dc(x)
dx
��2dx| {z }
Dq
s.t.Z xmax
xmin
dc(x)
dxdx = 2xmax
Lagrange Multiplier (�nd dc�(x)/dx = g �)
g � = argmin
�Z xmax
xminp(x) (g)�2 dx + λ
�Z xmax
xmingdx � 2xmax
��Optimal Companding Law c�(x)
g � =dc�(x)
dx=
xmaxR xmax0
3pp(x)dx
3
qp(x)
D�q '2
3L2
�Z xmax
0
3
qp(x)dx
�2= σ2f 2
�2Rε2
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 22 / 61
Lecture 6 Scalar and Vector Quantization MMAE Scalar Quantizer
Minimum Mean Absolute Error Quantizer
Distortion Measure
d(F ,Q(F )) � jF �Q(F )j
Objective
(b�, g�) = arg minfb,gg
∑l2L
Z bl
bl�1jf � gl j p(f )df
!| {z }
J(b,g)
whereb = (b0, b1, ..., bL)
T , g = (g1, g2, ..., gL)T
Necessary Conditions
rJ(b�, g�) = 0) (1) ∂J(b�, g�)/∂bl = 0(2) ∂J(b�, g�)/∂gl = 0
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 23 / 61
Lecture 6 Scalar and Vector Quantization MMAE Scalar Quantizer
Minimum Mean Absolute Error Quantizer
Solution
(1) ∂J(b�, g�)/∂bl = jb�l � g �l j p(b�l )� jb�l � g �l+1j p(b�l ) = 0
(2) ∂J(b�, g�)/∂gl =Z gl
bl�1p(f )df �
Z bl
glp(f )df = 0 (Appendix)
Nearest Neighbor Condition (Same as MMSE Quantizer)
jb�l � g �l j p(b�l )� jb�l � g �l+1j p(b�l ) = 0) b�l =g �l + g
�l+1
2
Generalized Centroid ConditionZ g �l
b�l�1
p(f )df =Z b�l
g �l
p(f )df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 24 / 61
Lecture 6 Scalar and Vector Quantization Optimal Scalar Quantizer
Optimal Scalar Quantizer
Nearest-Neighbor Condition (same as MMSE case)
B�l = ff : d(f , gl ) � d(f , gl 0), 8l 0 6= lg
Generalized Centroid Condition3
g �l = argminglE (d(F , g)jF 2 Bl )
Remarks
Nearest-Neighbor condition is NOT a�ectedCentroid SHALL be adapted to Distortion MeasureSource distribution MUST be knownReconstruction level is indexed by a �xed-length code
3Solution depends on distortion measureWen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 25 / 61
Lecture 6 Scalar and Vector Quantization Optimal Scalar Quantizer
Lloyd-Max Algorithm
Optimal quantizer design based on training data
Applicable when source distribution is unknownUse sample averages to replace expectations
Update reconstruction and boundary values iteratively
1 Initialization
Choose initial reconstruction valuesCalculate initial boundary values (Nearest Neighbor)Calculate initial distortion
2 Iterations
Find new reconstructions (Centroid, Sample Mean)Find new boundaries (Nearest Neighbor)Calculate new distortionRepeat till distortion converges
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 26 / 61
Lecture 6 Scalar and Vector Quantization Optimal Scalar Quantizer
Lloyd-Max Algorithm
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 27 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Entropy Constrained Optimal Scalar Quantizer
Minimize Quantizer Output Entropy s.t. Quantizer Distortion in MSE
minHQ = �∑l2LP(Bl ) log2 P(Bl )| {z }
Quantizer Output Entropy
s.t. Dq =x2max3L2
Z xmax
xminp(x)
�dc(x)
dx
��2dx| {z } = D
Quantizer Distortion with L Levels (High Rate Approx.)
Assumptions
High Rate ApproximationAn arbitrary number of quantization levels L
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 28 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Entropy Constrained Optimal Scalar Quantizer
Quantizer Output Entropy (in terms of c(x))
HQ = �∑l2LP(Bl ) log2 P(Bl ) (3)
' �∑l2Lp(gl )∆l log2 (p(gl )∆l )
' �Z xmax
xminp(x) log2 p(x)dx| {z }
h(X )=E (� log2 p(x)) di�erential entropy
+ log2L
2xmax+Z xmax
xminp(x) log2
dc(x)
dxdx
Optimal quantizer is equivalent to �nding g � = dc�(x)/dx such that
minZ xmax
xminp(x) log2
dc(x)
dxdx + λ
Z xmax
xminp(x)
�dc(x)
dx
��2dx �D 0
!Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 29 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Entropy Constrained Optimal Scalar Quantizer
Uniform Quantizer attains minimum output entropy regardless of PDF
dc�(x)
dx=p2λ = Constant
c�(x) = x given c(xmax) = xmax, c(0) = 0 (4)
Optimal Output Entropy (From Eqs. (3) & (4))
H�Q(x) = h(X )� log∆ = h(X )� 12log2 (12Dq)
Information-theoretical Interpretation
h(X ) average information at quantizer inputlog2 ∆ = h(Q) average information loss due to quantization
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 30 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with Rate-Distortion Bound
Memoryless Non-Gaussian Sources (No closed-form D(R))
LD(R)| {z }Lower Bound
� D(R) � D(R)G| {z }Upper Bound
where distortion D is in MSE and
LD(R) = (2πe)�12�2[R�h(X )],
LR(D) = h(X )� 12log2 2πeD
For a given distortion
H�Q(x)�L R(D) = 0.255bits
Lower e�ciency is caused by scalar (or memoryless) quantization
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 31 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with MMSE Scalar Quantizer
For a given distortion, maximum bit rate reduction
maxf∆Rg = 1
2log2
σ2fDq
ε2 �H�Q(x) =1
2log2
�12σ2f ε2
�� h(X )
For a given rate, maximum SNR gain
maxf∆SNRg = SNREC � SNRMMSE = 6.02maxf∆Rg dB
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 32 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with MMSE Scalar Quantizer
maxf∆RgRate reduction with entropy constrained quantizerA higher number of quantization levels
∆Rpdf�optRate reduction by entropy encoding MMSE quantizer outputsA �xed number of quantization levels
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 33 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Comparison with MMSE Scalar Quantizer
Gaussian Source Even L vs. Odd L (Entropy Constrained)
(MMSE Scalar Quantizer L=8)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 34 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Summary
∆Rpdf�optNOT the best value achievable by entropy coding
maxf∆Rg = (3/2)∆Rpdf�optAchieved with uniform quantizer and more quantization levelsAdditional quantization levels are used for outer part of PDFFall short of R-D bound by 0.255bits (1.53dB) at high ratesApproach R-D bound at low rates
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 35 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Optimal Scalar Quantizer
Vector Quantizer
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 36 / 61
Lecture 6 Scalar and Vector Quantization Vector Quantization
Vector Quantization
Why Vector Quantization?
Samples in a block are correlatedSome block patterns are more likely than others
Parameters
Quantization Levels LReconstruction Vectors (Codewords) gl , l = 1, 2, ..., LPartition Regions Bl , l = 1, 2, ..., LQuantizer Mapping Function
gl = Q(f) if f 2 Bl
wheref = [f1, f2, ..., fN ]
T , gl = [gl ;1, gl ;2, ..., gl ;N ]T
Bits/Sample R = 1N dlog2 Le
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 37 / 61
Lecture 6 Scalar and Vector Quantization Nearest-Neighbor Vector Quantizer
Nearest-Neighbor Vector Quantizer
Quantized vector gl determined by Nearest Neighbor criterion
Complexity (increases exponentially with vector size N)
Operations NL (or N2NR ), storage NL (or N2NR )
Bl = ff 2 RN : dN(f, gl ) � dN(f, g0l ), 8l 0 6= lg
dN(f, gl ) =1
N ∑N
n=1(fn � gl ;n)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 38 / 61
Lecture 6 Scalar and Vector Quantization Lattice Vector Quantizer
Lattice Vector Quantizer
Realization of Uniform Vector Quantizer
All partitions have same shape and size (good for uniform source)Closed-form quantizer mapping function (low complexity)
Reconstruction Vectors gl are Lattice Points (or Lattice Coset)
Lattice Λ (de�ned by a set of basis vectors fvng)
gl =N
∑n=1
ml ;nvn = [V]ml ,
whereml 2 Zn
[V] = [v1, v2, ..., vN ] is Lattice Generating Matrix
Determination of Quantized Vector
1 m =[V]�1f, where m 2 Rn is a real index vector2 Evaluate distortions of bm = dme or bmc, where bm 2 Zn
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 39 / 61
Lecture 6 Scalar and Vector Quantization Lattice Vector Quantizer
Quantizer Distortion
Vector Quantizer Distortion (Same as Scalar Quantizer)
Dq = E (dN(�!F ,Q(�!F )))
=ZBpN(f)dN(f,Q(f))df
=L
∑l=1
P(Bl )Dq,l
where
Dq,l =ZBlpN(f jf 2 Bl )dN(f, gl )df
Example: Lattice Quantizer for Uniform Distribution
Dq =L
∑l=1
P(Bl )Dq,l = Dq,l and Dq,l =1
jdet[V]j
ZBldN(f, 0)df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 40 / 61
Lecture 6 Scalar and Vector Quantization Lattice Vector Quantizer
Lattice Vector Quantizer
Rectangular vs. Hexagonal
Better space packing makes VQ outperform SQ even with i.i.d source
dN (f, gl ) = maxf2Bl
dN (f, gl )
Rectangular Lattice: 2 uniform scalar quantizers
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 41 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Optimal Vector Quantizer
Objective
(B�1 ,B�2 , ...,B�L; g�1, g�2, ..., g�L)= argminE (d(
�!F ,Q(�!F )))
= argmin
�ZBdN(f,Q(f))p(f)df
�= argmin
�∑P(Bl )
ZBldN(f,Q(f))p(f jf 2 Bl )df
�Optimal Solution
1 Given�g�1, g
�2, ..., g
�L
�, what would be optimal (B1,B2, ...,BL)?
2 Given�B�1,B�2, ...,B�L
�, what would be optimal (g1, g2, ..., gL)?
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 42 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Optimal Vector Quantizer
Given (g�1, g�2, ..., g
�L), what would be optimal (B1,B2, ...,BL)?
Q�(f) = arg minQ(f)=fg�1 ,g�2 ,...,g�Lg
�ZBdN(f,Q(f))p(f)df
�= arg min
Q(f)=fg�1 ,g�2 ,...,g�LgdN(f,Q(f))
B�l = ff : dN(f, g�l ) � dN(f, g�l 0), 8l 0 6= lg (Nearest Neighbor)
Given (B�1 ,B�2 , ...,B�L), what would be optimal (g1, g2, ..., gL)?
g�l = argming
�∑P(B�l )
ZB�ldN(f, g)p(f jf 2 B�l )df
�= argmin
g
ZBldN(f, g)p(f jf 2 B�l )df
= argmingE (dN(
�!F , g)j�!F 2 B�l ) (Centroid)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 43 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Optimal Vector Quantizer
Nearest-Neighbor and Centroid are Necessary Conditions (Candidates)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 44 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Lloyd-Max Algorithm
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 45 / 61
Lecture 6 Scalar and Vector Quantization Optimal Vector Quantizer
Lloyd-Max Algorithm
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 46 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Vector Quantizer
Entropy Constrained Vector Quantizer
ObjectiveminDq s.t. �∑
l2LP(Bl ) log2 P(Bl )| {z }Entropy Rate
= RN
Solution (Necessary Conditions)
Lagrangian
(B�1,B�2, ...,B�L; g�1, g�2, ..., g�L)
= argmin
Dq + λ�
�∑l2LP(Bl ) log2 P(Bl )� RN
!!| {z }
J(B1,B2,...,BL;g1,g2,...,gL)4
λ� must be chosen such that
�∑l2LP(B�l ) log2 P(B�l ) = RN
4Lloyd-Max Algorithm is still applicable by replacing Dq with J(�)Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 47 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Vector Quantizer
Gaussian-Markov Source + 8D VQ
http://www.stanford.edu/class/ee368b/Handouts/06-Quantization.pdf
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 48 / 61
Lecture 6 Scalar and Vector Quantization Entropy Constrained Vector Quantizer
Memoryless Laplacian + 8D VQ
http://www.stanford.edu/class/ee368b/Handouts/06-Quantization.pdf
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 49 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Appendix
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 50 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Properties of MMSE Quantizer
G is unbiased estimator of F
E (G) = ∑l
P(Bl )gl
= ∑l
P(Bl )�Z
Blf p(f jf 2 Bl )df
�= ∑
l
ZBlf p(f )df
=ZBf p(f )df = E (F )
Q = F � G is of zero mean
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 51 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Properties of MMSE Quantizer
G is orthogonal to Q
E (GQ) = E (G(F � G)) = ∑l
P(Bl )ZBlgl (f � gl ) p(f jf 2 Bl )df| {z }
=0
= 0
For an MMSE quantizer
gl = E (f jf 2 Bl ) =ZBlf p(f jf 2 Bl )df
ThusZBlgl (f � gl ) p(f jf 2 Bl )df = glE (f jf 2 Bl )| {z }
=gl
� g2l = 0, 8l 2 L
Note that E (F (F � G)) = E (F2)� E (FG) = E (F2)� E (G2) 6= 0Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 52 / 61
Lecture 6 Scalar and Vector Quantization Appendix
Properties of MMSE Quantizer
Reduced signal variance
σ2G = σ2F � σ2Q
By de�nition
σ2Q = E ((F � G)2)= E (
�(F�µF )�
�G�µG
��2)
= σ2F + σ2G � 2E ((F�µF )�G�µG
�)
= σ2F + σ2G � 2(E (FG)�µFµG)
= σ2F + σ2G � 2(E (G2)�µ2G)
= σ2F � σ2G
where µF = µG = E (G) and E (FG) =E (G2)
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 53 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer High Rate Approximation
Quantizer Distortion
Dq ' ∑l2L
P(Bl )∆2l12
= ∑l2L
p(g �l )∆3l
12= ∑
l2L
α3l12
where
αl =3
qp(g �l )∆l
Observe that
∑l2L
αl = ∑l2L
3
qp(g �l )∆l '
Z fmax
fmin
3
qp(f )df = constant
MMSE Scalar Quantizer
min ∑l2L
α3l12s.t. ∑
l2Lαl = C
) α�l = constant =1
L
Z fmax
fmin
3
qp(f )df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 54 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer High Rate Approximation
Given
α�l =1
L
Z fmax
fmin
3
qp(f )df
Quantizer Distortion
Dq ' ∑l2L
(α�l )3
12= L
(α�l )3
12
=1
L21
12
�Z fmax
fmin
3
qp(f )df
�3= 2�2R
1
12
�Z fmax
fmin
3
qp(f )df
�3where L = 2R
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 55 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer High Rate Approximation
Given F 0 = F/σf is an unit variance source
P(F 0 � f 0)| {z }CF0 (f
0)
= P(F � σf f0)| {z }
CF (σf f 0)
) d
df 0CF 0(f
0) =d
df 0CF (σF f
0)
) pF 0(f0) = σf pF (σf f
0) or pF (f ) =1
σfpF 0(
f
σf)
Quantizer Distortion
Dq ' 2�2R
12
�Z fmax
fmin
3
qpF (f )df
�3=2�2R
12
Z fmax
fmin
3
s1
σfpF 0(
f
σf)df
!3= σ2f 2
�2Rε2
where ε2 = 112
�R fmax/σffmin/σf
3ppF 0(f )df
�3Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 56 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer of Various Sources
Given a source F 0= (F � µF ) /σF of zero mean and unit variance
P(F 0 � f 0)| {z }CF0 (f
0)
= P(F � σF f0 + µF )| {z }
CF (σF f 0+µF )
) d
df 0CF 0(f
0) =d
df 0CF (σF f
0 + µF )
) pF 0(f0) = σFpF (σF f
0 + µF ) or pF (f ) =1
σFpF 0(
f � µFσF
)
where
pF 0(f0) and pF (f ) are PDF of F 0 and F , respectively
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 57 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer of Various Sources
Optimal Scalar Quantizer for F 0
(1) ebl = egl+egl+12
(2) egl = E (F 0jF 0 2 eBl ) = R eblebl�1 f 0 pF0 (f 0)df 0R eblebl�1 pF0 (f 0)df 0Optimal Scalar Quantizer for F
(1) bl =gl+gl+1
2 = σFebl + µF
(2) gl = E (FjF 2 Bl ) =R blbl�1
f pF (f )dfR blbl�1
pF (f )df= σFegl + µF
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 58 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMSE Scalar Quantizer of Various Sources
Let
f 0 =f � µF
σFAssume ebl = bl � µF
σFThen optimal gl for F can be obtained as follows:
gl = E (FjF 2 Bl ) =R blbl�1
f pF (f )dfR blbl�1
pF (f )df=
R blbl�1
f 1σFpF 0(
f�µFσF
)dfR blbl�1
1σFpF 0(
f�µFσF
)df
=
R eblebl�1 (σF f 0 + µF ) pF 0(f0)df 0R eblebl�1 pF 0(f 0)df 0 = σFegl + µF (5)
From Eq. (5), the assumption ebl = (bl � µF ) /σF is justi�ed
bl = (gl + gl+1) /2 = σFebl + µFWen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 59 / 61
Lecture 6 Scalar and Vector Quantization Appendix
MMAE Scalar Quantizer
Observe thatZ bl
bl�1jf � gl j p(f )df
=Z gl
bl�1(gl � f ) p(f )df +
Z bl
gl(f � gl ) p(f )df
= gl
Z gl
bl�1p(f )df �
Z gl
bl�1f p(f )df +
Z bl
glf p(f )df � gl
Z bl
glp(f )df
Then
∂
∂glJ(b�, g�) =
∂
∂gl
�Z bl
bl�1jf � gl j p(f )df
�=
Z gl
bl�1p(f )df �
Z bl
glp(f )df
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 60 / 61
Lecture 6 Scalar and Vector Quantization Appendix
References
1 Y. Wang, et. al - Video Processing and Communications
2 Jayant, N. S., et. al - Digital Coding of Waveforms
3 B. Giord, http://www.stanford.edu/class/ee368b/Handouts/
Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL May 2008 61 / 61