A i - M e i H u a n g A n d Tr u o n g N g u y e nI m a g e   p r o c e s s i n g , 2 0 0 6 I E E E i n t e r n a t i o n a l c o n f e r e n c e   on  

Motion vector processing based on residual energy information for

motion compensated frame interpolation

INTRODUCTION

• Low bandwidth requirements, i.e. video telephony• by skipping frames• low frame rate video usually

results in motion jerkiness

• Motion-compensated frame interpolation (MCFI)• Directly uses the received MVs• Suffer from annoying artifacts such as blockiness and ghost effect.• Motion re-estimation is not suited for mobile devices

Video in encoder

decoderReconstructed frame

Bit stream

Video coding/compression

Motion compensated frame interpolation

• In MCFI methods, a skipped frame is often bi-directionally interpolated • from its two neighboring reconstructed frames .• by using the received MVs of the second frame.

MV is not always reliable!

The correlation between motion vectorreliability and residual energy.

• Macroblock(MB) contain areas of

different motion.• Encoder favors

the MV that can represent most ofthe region

• The prediction residual will be generated and encoded.• Correct those unreliable MVs for frame interpolation• Avoid using those unreliable MVs to correct other MVs.

The proposed method

• Based on residual energy associated with each motion vector

• iterative process, stops when the MAP is no longer changed, or reaches a pre-defined maximum iteration, max_ite.

(1)motion vector classification based on residual energy

• Classify MVs to 3 Groups.• Calculate every 8x8 block’s

residual energy Em,n

• By taking the sum of the absolute value ofeach reconstructed prediction error for each pixel.• Em,n<threshold MV will be classified as

reliable and place into first group G1.• Em,n>=threshold MV will be classified as

unreliable and place into first group G3.• Unreliable MV’s neighboring MVs within the same MB will be

classified as possibly unreliable into the second group G2

(base unit)16*16 MB=>four 8*8 blocks

(2) motion vector correction

• Works on those unreliable MVs Correcting from neighbor reliable MV

• A residual-energy constrained median filter (RECMF) is used in this process and defined as

• S contains the neighboring MVs centered at v*m,n.• Select a new MV only from its neighboring reliable MVs.

• vm,n itself will be excluded from the candidates.• we prevent unreliable MVs to be used to correct other

unreliable MVs.

(3)motion vector similarity check

• Ensure v*m,n is not identical or similar to vm,n

• Remain unreliable

• Use angle variance by taking inner product of the two vectors.• < threshold => two MVs is similar ,fail in the check => bm,n still in G3• Pass in the check => bm,n take in G1

(4)motion vector re-sampling and smoothing

• In [7], each 8 x 8 block is further broken into four 4 x4 sub-blocks.• The MVs of these four sub-blocks can be obtained

simultaneously by minimizing a smoothness measure

, which is defined in the following.

• For example:

[7] G. Dane and T. Q. Nguyen, "Smooth motion vector resamplingfor standard compatible video post-processing," Proc. AsilomarConf: Signals, Systems and Computers, 2004.

SIMULATIONS

• two video sequences, FOREMAN and SILENT, • CIF frame resolution • 30 frame per second (fps).• both encoded using H.263 but even frames are skipped

by the encoder.• fix quantization parameter (QP) values.

• Avg. bitrate• FOREMAN is 240.3 Kbps • SILENT is184.59 Kbps