block loss recovery techniques for image communications jiho park, d-c park, robert j. marks, m....

Block Loss Recovery Techniques for Image Communications

Jiho Park, D-C Park, Robert J. Marks, M. El-Sharkawi

The Computational Intelligence Applications (CIA) Lab.

Department of Electrical Engineering

University of Washington

May 29, 2002

2

Projections based Block Recovery – Motivation

Conventional Algorithms use information of all surrounding area. Using only highly correlated area

3

Alternating Projections is projecting between two or more convex sets iteratively.

Alternating Projections

Converging to a common point

4

Projections based Block Recovery – Algorithm

2 Steps Pre Process : 1) Edge orientation detection

2) Surrounding vector extraction

3) Recovery vector extraction

Projections : 1) Projection operator P1

2) Projection operator P2

3) Projection operator P3

5

Edge orientation in the surrounding area(S) of a missing block(M). In order to extend the geometric structure to the missing block.

Simple line masks at every i, j coordinate in surrounding area(S) of the missing block(M) for edge detection.

Pre Process 1 –Edge Orientation Detection

121

121

121

vL

111

222

111

hL

Horizontal Line Mask Vertical Line Mask

6

Pre Process 1 – Edge Orientation Detection

Responses of the line masks at window W :

Total magnitude of responses :

Th > Tv ; Horizontal line dominating area

Th < Tv ; Vertical line dominating area

987

654

321

www

www

www

W987654321 w-w-w-w2w2w2w-w--w hR

987654321 w-w2w-w-w2w-w-w2-w vR

,||T S

hh R S

vv R ||T

7

Pre Process 2 – Surrounding Vectors

Surrounding Vectors, sk, are extracted from surrounding area of a missing block by N x N window.

Each vector has its own spatial and spectral characteristic. The number of surrounding vectors, sk, is 8N.

}W),(),,(:{ jijixxks

8

Pre Process 3 – Recovery Vector Recovery vectors are extracted to restore missing pixels. Two positions of recovery vectors are possible according to the

edge orientation.

Recovery vectors consist of known pixels(white color) and missing pixels(gray color).

The number of recovery vectors, rk, is 2.

}W),(),,(:{ jijixxkr

Vertical line dominating area Horizontal line dominating area

9

Projections based Block Recovery –Projection operator P1

Recovery vectors, ri, for i = 1, 2

Surrounding vectors, sj , for j = 1 ~ 8N

Surrounding vectors, S, form a convex hull in N2-dimensional space

Recovery vectors, R, are orthogonally projected onto the line defined by the closest surrounding vector, si, j : Projection Operator P1.

10

Projections based Block Recovery –Projection operator P1

Projection operator P1 :

Convex hull (formed by surrounding vectors, containing information of local image structure)

11

Projections based Block Recovery –Projection operator P1

Surrounding vectors, sj , for j = 1 ~ 8N Recovery vectors, ri, for i = 1, 2

The closest vertex, sdi , from a recovery vector, ri.

or equivalently in DCT domain,

P1 :

Njiford jij

i 81,21||||minarg sr

Njiford jij

i 81,21||||minarg SR

21,||||

,)(

2

ii

i

d di

idiiP S

R

RSRS

12

Convex set C2 acts as an “identical middle”.

Projection operator P2 :

Projections based Block Recovery –Projection operator P2

otherwise

nforFFC

o

n

ff

ff

:

L: maxmin2

otherwise

nFforF

nFforF

P

n

n

n

n

f

f

f

f L

L

max,max

min,min

2

13

Convex set C3 acts as a convex constraint between missing pixels and adjacent known pixels, (fN-1 fN).

where,

and is a N x N recovery vector in

column vector form.

Projections based Block Recovery – Projection operator P3

fN-1 fN

}||:{3 EC n gg

)}(....,),{( ,,10,0,1 NNNNNN ffffg

}....,,,{ 21 Nffff

Projection operator P3 :

otherwise

nEforE

nEforE

P

mn

nmn

nmn

mn

,

,1

,1

,3 L,

L,

f

gf

gf

f

14

Projections based Block Recovery –Iterative Algorithm

Missing pixels in recovery vectors are restored by iterative algorithm of alternating projections :

N x N windows moving :

ii fPPPf 3211

Vertical line dominating area Horizontal line dominating area

15

Projections based Block Recovery - Summary

Edge Orientation Detection

Surrounding Vector Extraction

Recovery Vector Extraction

Projection Operator P1

Projection Operator P2

Projection Operator P3

Iteration=I?

All pixels?

16

Simulation Results –Lena, 8 x 8 block loss

Original Image Test Image

17

Simulation Results –Lena, 8 x 8 block loss

Ancis, PSNR = 28.68 dB Hemami, PSNR = 31.86 dB

18

Simulation Results –Lena, 8 x 8 block loss

Ziad, PSNR = 31.57 dB Proposed, PSNR = 34.65 dB

19

Simulation Results –Lena, 8 x 8 block loss

Ancis

PSNR = 28.68 dB

Hemami

PSNR = 31.86 dB

Ziad

PSNR = 31.57 dB

Proposed

PSNR = 34.65 dB

20

Simulation Results – Each StepLena 8 x 8 block loss

(a)

(b)

(c)

21

Simulation Results –Peppers, 8 x 8 block loss

Original Image Test Image

22

Simulation Results – Peppers, 8 x 8 block loss

Ancis, PSNR = 27.92 dB Hemami, PSNR = 31.83 dB

23

Simulation Results – Peppers, 8 x 8 block loss

Ziad, PSNR = 32.76 dB Proposed, PSNR = 34.20 dB

24

Simulation Results –Lena, 8 x one row block loss

Original Image Test Image

25

Simulation Results –Lena, 8 x one row block loss

Hemami, PSNR = 26.86 dB Proposed, PSNR = 30.18 dB

26

Simulation Results –Masquerade, 8 x one row block loss

Original Image Test Image

27

Simulation Results –Masquerade, 8 x one row block loss

Hemami, PSNR = 23.10 dB Proposed, PSNR = 25.09 dB

28

Simulation Results –Lena, 16 x 16 block loss

Original Image Test Image

29

Simulation Results –Lena, 16 x 16 block loss

Ziad, PSNR = 28.75 dB Proposed, PSNR = 32.70 dB

30

Simulation Results –Foreman, 16 x 16 block loss

Original Image Test Image

Ziad, PSNR = 25.65 dB Proposed, PSNR = 30.34 dB

31

Simulation Results –Flower Garden, 16 x 16 block loss

Original Image Test Image

Ziad, PSNR = 20.40 dB Proposed, PSNR = 22.62 dB

32

Simulation Results – Test Data and Error

512 x 512 “Lena”, “Masquerade”, “Peppers”, “Boat”, “Elaine”, “Couple”

176 x 144 “Foreman” 352 x 240 “Flower Garden”

8 x 8 pixel block loss 16 x 16 pixel block loss 8 x 8 consecutive block losses

Peak Signal – Noise – Ratio

)|),(ˆ),(|

255log(10

1 1

2

2

N

i

M

j

jixjix

MNPSNR

33

Simulation Results – PSNR (8 x 8)

Lena Masqrd Peppers Boat Elaine Couple

Ancis 28.68 25.47 27.92 26.33 29.84 28.24

Sun 29.99 27.25 29.97 27.36 30.95 28.45

Park 31.26 27.91 31.71 28.77 32.96 30.04

Hemami 31.86 27.65 31.83 29.36 32.07 30.31

Ziad 31.57 27.94 32.76 30.11 31.92 30.99

Proposed 34.65 29.87 34.20 30.78 34.63 31.49

34

Simulation Results – PSNR (Row, 16 x 16)

(16 x 16) Lena Foreman Garden

Ziad 28.75 25.65 20.40

Proposed 32.70 30.34 22.62

(8 x Row) Lena Maskrd Peppers Boat Elaine Couple

Hemami 26.86 23.10 25.41 24.54 26.87 24.30

Proposed 30.18 25.09 28.31 26.06 30.11 26.12