Source-Channel Prediction in Error Resilient Video Coding

Hua Yang and Kenneth RoseSignal Compression Laboratory

ECE Department

University of California, Santa Barbara

7/8/03 ICME 2003 2

Outline

Introduction

Source-channel prediction

Simulation results

Conclusions

7/8/03 ICME 2003 3

Introduction Existent error resilient approaches on the prediction

mechanism Slice coding

limit prediction within certain non-overlapping spatial regions Video redundancy coding

Multiple independently predicted “threads” Multi-frame motion compensation

Multiple reference frames for prediction

Feature in common: assume the same underlying conventional prediction framework

Framework: separate source-channel coding Prediction: past encoder reconstructed frames Motion estimation criterion: minimum prediction error

7/8/03 ICME 2003 4

Introduction

Via considering packet loss effects during encoding, joint source-channel coding usually achieves better error resilience than that of separate coding.

Our proposed approach Prediction is based on expected decoder reconstruction of the previous

frames. Novelty

Unlike all the other existent error resilient prediction schemes and all the other existent source-channel coding schemes, our proposed method is actually a source-channel prediction scheme.

7/8/03 ICME 2003 5

Introduction0 p1 p2 p3

pi packet loss rate of video packet i.

p1 p2

p2

p3

p3

p3

p3

1-p1 1-p2 1-p3

1-p2

1-p3

1-p3

1-p3

Encoder reconstruction, i.e. “best possible” decoder reconstruction: quantization loss only.

Other possible decoder reconstructions:different transmission loss patterns.

Expected decoder reconstruction:quantization loss & transmission loss.

7/8/03 ICME 2003 6

Introduction Expected decoder reconstruction

Encoder’s estimate of the decoder reconstruction. Given the packet loss rate, it can be accurately computed with the

ROPE method.

Recursive optimal per-pixel estimate (ROPE)

Basic idea: ROPE accurately computes these unknown quantities in a recursive manner for all

the pixels of every frame. Accurate & Low complexity Frequently used to estimate end-to-end distortion in various RD optimization

scenarios. Now we use these expectations for source-channel prediction.

})~

{(}~

{2)(})~

{( 222 in

in

in

in

in

in

in fEfEffffEd

Random variableunknown

7/8/03 ICME 2003 7

Source-channel Prediction

Conventional prediction

Source-channel prediction

Prediction residue

jin

in ff

1ˆ

jin

in fEf

1~

in

in

in ffres

infi

nf Original and predicted values

For pixel i in frame n:

jnf 1

~

jnf 1

ˆ

Encoder and decoder reconstruction values of pixel j in frame n-1 to predict pixel i in frame n.

inres Prediction error to be quantized

7/8/03 ICME 2003 8

Source-channel Prediction Source-channel prediction is the optimal prediction in

the sense of minimum MSE end-to-end distortion.

Pending problem: motion estimation criterion ?

Criterion in the conventional scheme

jinfE 1

~

MBi

mvin

in

mvMBi

in

in

mvffff

2

12 ˆminmin

MBi

mvin

in

mvfEf

2

1 }~

{minCriterion I

Constant value:

Not the actual predictor of the decoder ji

ni

n fEf 1

~Plug in:

7/8/03 ICME 2003 9

Source-channel Prediction

Pending problem: motion estimation criterion? (cont.)

Criterion II

MBi

mvin

in

mvffE }~

{min2

1

Random variable:

Actual predictor of the decoder

Criterion II is superior than Criterion I in that it explicitly accounts for the randomness of the decoder’s actual predictor.

7/8/03 ICME 2003 10

Source-channel Prediction Another interpretation of Criterion II

DR

mv

MBi

mvin

mvin

mvin

in

mv

MBi

mvin

in

mv

DDp

fEfEfEf

ffE

)1(min

}~

{}){(}~

{min

~min

122

12

1

21

MBi

in

inR ffD

2

MBi

in

inD ffED

2~

While Criterion II considers the properly weighted impacts of both DR and DD , in contrast, Criterion I only considers DR . In this sense, Criterion II is more “comprehensive”.

7/8/03 ICME 2003 11

Simulation Results Simulation conditions

H.263+ video codec

System performance: average luminance PSNR 50 different packet loss patterns

Testing scenarios No INTRA Updating

Periodic INTRA Updating

For packet loss rate p, coding a MB in INTRA mode once for every 1/p frames.

R-D optimized INTRA Updating

For each MB, select its coding mode as INTER or INTRA with the R-D criterion.

7/8/03 ICME 2003 12

Foreman (300kbps)

13

18

23

28

33

1 51 101 151 201 251 301 351

Frame No.

PS

NR

(dB

)

EP SCP_CI SCP_CII

Foreman (200kbps)

23

24

25

26

27

28

29

5 10 15 20 25 30

Packet Loss Rate (%)

PS

NR

(d

B)

EP SCP_CI SCP_CII

Foreman (200kbps)

24

25

26

27

28

29

30

31

5 10 15 20 25 30

Packet Loss Rate (%)

PS

NR

(d

B)

EP SCP_CI SCP_CII

(a) No INTRA updating ( p = 10%)

(b) Periodic INTRA updating. (c) RD optimal INTRA updating.

7/8/03 ICME 2003 13

Simulation Results Observations

The proposed “SCP_CII” method consistently offers the best performance, which proves our previous analysis.

When INTRA updating is more effectively performed, smaller gains are achieved by “SCP_CII” over “EP”. Hence, the gain depends on how much damage of packet loss is not accounted for in the conventional scheme.

Similar results also hold for other testing sequences, e.g., carphone, miss_am, salesman, etc.

7/8/03 ICME 2003 14

Demo

Conventional prediction based on encoder reconstruction

(PSNR = 25.06dB)

Foreman, QCIF, 30f/s, 300kb/s, packet loss rate = 10%, periodic Intra update.

Source-channel predictionbased on expected decoder reconstruction

(PSNR = 26.72dB)

7/8/03 ICME 2003 15

Conclusions

Novelty: the proposal of further enhancement of error resilience via fundamental modification of the conventional prediction structure.

Source-channel prediction based on expected decoder reconstruction, which uses ROPE to get accurate estimate of decoder quantities.

In spite of the loss in source coding gain due to the lower source prediction quality, our scheme achieves better overall R-D tradeoff than the conventional scheme.

We identify the subtle points in selecting the motion estimation criterion, and shows that it is advantageous to use the criterion of minimizing the expected prediction error.

7/8/03 ICME 2003 16

Thanks!