School of Electrical Engineering and Computer Science Kyungpook National Univ.

A New Edge-Adaptive Demosaicing Algorithmfor Color Filter Arrays

Image and Vision ComputingVol. 25, No. 9, 2007

Chi-Yi Tsai and Kai-Tai Song

Presented by Jung-Yul Choi

Abstract

Proposed algorithm– New edge-adaptive demosaicing algorithm (EADA)

• Reduce color artifacts in demosaiced images from color filter array (CFA)

– Aliasing error

• Design edge-adaptive filtering and post-processing schemes• Green channel interpolation

– Using any of existing image interpolation methods– Presenting new adaptive interpolation

– Experimental results of EADA• Outperforms recently proposed demosaicing techniques

2 / 34

Introduction

Digital color images– Obtain by interpolating output from color filter array

Color filter array (CFA)– A set of spectrally selective filters– Sensor pixel

• One of three primary color components

– Rendering full color image• Necessity of CFA demosaicing

– Drawback of single-channel algorithms» Color artifacts and blurring around sharp edges

3 / 34

Previous work

High cross-correlation between color channels– Use of inter-channel color ratios

• Smooth hue transition algorithms– Assumption

» Hue don’t change abruptly between neighboring pixels

• Interpolating missing color values– Exploiting color ratios between R and G, B and G

– Use of inter-channel color differences• R–G and B–G

– Outperforming single-channel algorithms– Still unsatisfactory demosaicing results

4 / 34

Edge-directed interpolation approach– Perform interpolation along image edges– Prevent interpolation across edges– Procedure

• Analyzing spatial correlation of local image neighborhood• Selecting suitable interpolation direction

Use both two demosaicing approaches– Kimmel’s hybrid demosaicing method

• Combining edge-directed interpolation with smooth hue transition algorithm in iterative process

– Li and Orchard• Proposing edge-directed interpolation scheme to

interpolating inter-channel color differences

5 / 34

Another methods– Gunturk et al.

• Use edge-directed interpolation scheme– Estimating missing color values in green channel

• Use alternating projection scheme– Estimating missing color values in red and blue channels

– Lu and Tan• Two steps for improved hybrid CFA demosaicing method

– Interpolation step» Rendering full-color images

– Post-processing step» Suppressing visible demosaicing artifacts

6 / 34

Second and third class algorithms– Produce high quality visual results– Edges in regions of fine details

• Intoducing undesirable errors– Aliasing error in high frequency regions

Proposed method– New edge-adaptive CFA demosaicing algorithm

• Reproducing color values– Making high-frequency components of red and blue

channels similar to green channel

7 / 34

Color-difference approach to demosaicing

Bayer pattern– Most used CFA pattern– Common problem in demosaicing

• Visible color artifacts in high-frequency regions– Aliasing error of high-frequency components in green

channel less than that in red and blue

Fig. 1. Bayer color filter array patter.

8 / 34

High inter-channel correlation – R, G, and B at fine texture and edges

• Appearing very similar pattern

– High-frequency regions• Similar in all three channels and close to high-frequency

regions of green channel

– Assumption• Object boundaries are same in all three color channels

9 / 34

Validation of assumption– Flowchart for demonstrating

Fig. 3. Flowchart for demonstrating the assumption of color-difference model.

0 0lowpass highpass

0 0

1 if ( , ) 0 if ( , )( , ) and ( , )

0 if ( , ) 1 if ( , )D u v D D u v D

H u v H u vD u v D D u v D

≤ ≤ = = > >

where D0 is nonnegative quantity (D0 =128), andD(u, v) is distance from point (u, v) to origin of frequency plane.

10 / 34

– Use 24 natural images for demonstrating

Fig. 2. Test images used in the experiment.11 / 34

– MSE comparison results• Adding high-frequency regions of green plane to low-pass

filtered red and blue planes– Reducing MSE

• Assumption is validatedTable 1. Comparison of mean squared error at each step in Fig. 3.

12 / 34

– Study based on assumption• Reducing color artifacts in high-frequency regions

– Adding high-frequency information of green channel to other color channels

• Achieving by utilizing color-difference model

,

, and

d d d

l h

d d d

l h

d d d

l h

F R F R F R

F G F G F G

F B F B F B

= +

= +

= +

(1)

where [Rd Gd Bd] are three color planes of demosaiced image,F[·] is 2-D discrete Fourier transform, andsubscripts l and h stand for low-frequency and high-frequency components, respectively.

13 / 34

– Color-difference models of demosaiced image

andd d d dg gR R G B B G= − = −

{ }{ }

,d dg g l ld d

g g l l

F R L F R F R F G

F B L F B F B F G

= = −

= = −

(3)

(2)

,d d d dg l hd d d d

g l h

F R F R G F R F G

F B F B G F B F G

= + = +

= + = +

where L{·} is linear low-pass filtering process,and are low-frequency regions of color differences and , andand are new red and blue planes of demosaiced image.

gR gB gR gBdR dB

(4)

14 / 34

Proposed edge-adaptive demosaicing algorithm

Simplified CFA demosaicing– Interpolation step

• Obtaining full color demosaiced image

– Post-processing step• Remove false colors or color artifacts

Fig. 4. Simplified CFA demosaicing procedure in digital cameras.

15 / 34

Edge-adaptive low-pass filtering– Color difference planes

• Initial demosaiced image– Green channel

» Fully recovered by using existing image interpolation– Red and blue channels

» Obtaining by use bilinear interpolation

andd d d dg i i g i iR R G B B G= − = −

where are three color planes of initial demosaiced image.d d di i iR G B

(5)

16 / 34

– Two sub-steps of filtering procedure• Edge-adaptive low-pass filtering of red (blue) over original

blue (red) pixels• Edge-adaptive low-pass filtering of red (blue) over original

green pixels

(a) (b)

Fig. 5. (a) The red value on blue pixel and (b) the red value on green pixel of central pixel to be estimated.

17 / 34

– Determination of color difference value Rg

• Edge indicator of red and blue cases

• Edge indicator of green case

1 1 2 2 3 3 4 4

1 2 3 4

ˆˆˆˆa g a g a g a g

ga a a a

e R e R e R e RR

e e e e+ + +

=+ + +

(6)

where are color-difference adjusted values, andare edge indicators corresponding to each color-difference

adjusted value.

1 4ˆˆ ~g gR R

1 4~a ae e

1 3 5 1 3 4 2 4 2 6

1 3 1 3 7 4 2 8 4 2

1 22 2

2 2 2 2 2 2 2 2

3 42 2

2 2 2 2 2 2 2 2

1 1, ,1 1

1 1, and1 1

g g g g g g g g g g

g g g g g g g g g g

a aR R R R R R R R R R

a aR R R R R R R R R R

e e

e e

− − + − − +

− − + − − +

= =+ + + +

= =+ + + +

3 1 3 1 5 2 4 6 2 4

3 1 7 3 1 2 4 2 4 8

1 22 22 2 2 2

3 42 22 2 2 2

1 1, ,1 1

1 1, and1 1

g g g g g g g g g g

g g g g g g g g g g

a aR R R R R R R R R R

a aR R R R R R R R R R

e e

e e

− − + − − +

− − + − − +

= =+ + + +

= =+ + + +

(7.a)

(7.b)

18 / 34

• Color-difference adjusted values– Assumption

» Difference of neighboring color-difference values along interpolation direction is constant

– Relationships along right-up direction

– Combining (8) and (9)

– Other interpolation directions

(8)( ) ( )1 3 1 31 3

ˆˆ and

ˆˆg g g g g g

g g g g

R R R R R R

R R R R

− = − + −

− = −

4 2 1 32 2 3 3

2 44 4

ˆˆ , , and2 2

ˆ2

g g g gg g g g

g gg g

R R R RR R R R

R RR R

− −= + = +

−= +

(9)

where is missing color-difference value at locationˆgR gB

( )1 3 1 ˆ2g g g gR R R R− = − 3 11 1ˆˆ 2g g

g g g

R RR R R

−= + =

19 / 34

– Acquisition of full-red plane• Recovering spatial plane from color-difference plane

– Same procedure for blue plane• Acquisition of full-color demosaiced image

d di g iR R G= + (10)

20 / 34

Post-processing– Suppressing color artifacts of demosaiced image– Correction of green values

• Color-difference plane between green and red (blue)andd d d dr i i b i iG G R G G B= − = − (11)

d d di i iR G B where is color planes of full-color demosaiced image.

Fig. 6. Adaptive filtering of the red-green color difference plane.

21 / 34

• Estimation of color-difference value Gr8

81

1

, bjr j rj jj bkk

eG w G w

e==

= =∑∑

(12)

where ebk are edge indicators.

5 1 9 2 6 10

3 7 11 8 4 12

5 1 13 2 6 14

3 7 15 8 4 1

1 22 2 2 2 2

3 42 2 2 2 2

5 62 2 2 2 2

7 82 2 2 2

1 1, ,1 1

1 1, ,1 1

1 1, ,1 1

1 1, and1 1

r r r r r r r r

r r r r r r r r

r r r r r r r r

r r r r r r r

b bG G G G G G G G

b bG G G G G G G G

b bG G G G G G G G

b bG G G G G G G

e e

e e

e e

e e

− − − −

− − − −

− − − −

− − −

= =+ + + +

= =+ + + +

= =+ + + +

= =+ + + + 6

2

.rG−

(13)

22 / 34

– Post-processed green plane

– Post-processed red and blue planes

– Acquisition of post-processed demosaiced images• Repeating three times

( ) ( )2

d dr i b id

p

G R G BG

+ + += (14)

andd d d dgp i p gp i pR R G B B G= − = −

andd d d dp gp i p gp iR R G B B G= + = +

23 / 34

– Summary of proposed demosaicing algorithm

Fig. 7. Complete steps of the proposed edge-adaptive demosaicing algorithm.

24 / 34

Green channel adaptive interpolation

Green plane– Great influence on perceptual quality of image

Proposal– Nonlinear procedure for choosing interpolation direction

Fig. 8. Two cases of missing green value on the central pixel.25 / 34

Central missing green value G

1 1 3 3

1 3

2 2 4 4

2 4

1 1 2 2 3 3 4 4ˆˆ

1 3 2 4

ˆˆ2 4 1 3

1 1 2 2 3 3 4 4

ˆˆˆˆ if

if and

if and

ˆˆˆˆ otherwise

e e

e e

e e

e e

s s s s

w G w Ge e e ew w

w G w Ge e e ew w

e e e e

w G w G w G w G E T

E T w w w wG

E T w w w w

w G w G w G w G

++

++

+ + + < ≥ + > +=

≥ + > +

+ + +

where

are color-adjusted green values, T is threshold (T =10), andws1~ws4 and we1~we4 are associated weights when E

– Edge indicators in smooth regions

– Edge indicators in edge regions

9 4 10 2 5 1 6 21 3 5 1 1 2 2

11 1 12 3 7 2 8 32 4 6 2 2 2 2

2 13 4 14 3 9 4 101 3 3 7 3 2 2

1 16 3 15 42 4 4 8 4 2

11 1

12 1

13 1

14 1

,

,

, and

G G G G R R R R

G G G G R R R R

G G G G R R R R

G G G G R

s G G G G B B

s G G G G B B

s G G G G B B

s G G G G B B

e

e

e

e

− + − − + −

− + − − + −

− + − − + −

− + −

+ − + − + − + +

+ − + − + − + +

+ − + − + − + +

+ − + − + − + +

=

=

=

=11 1 12

2

.R R R− + −

1 3 5 1 3 1 1 3 5 1 9 4 6 2 10 2

2 4 6 2 4 2 2 4 7 2 11 1 8 3 12 3

1 3 1 3 7 1 3 3 2 13 3 9 4 14 4 10

2 4 2

11 1 2

12 1 2

13 1 2

14 1 2

,

,

, and

e G G G G G B B G G R R G G R R G G

e G G G G G B B G G R R G G R R G G

e G G G G G G G B B G G R R G G R R

e G G G

e

e

e

e

+ − + − + + − − + + − − + + − − +

+ − + − + + − − + + − − + + − − +

+ − + − + + − − + + − − + + − − +

+ − + −

=

=

=

=4 8 2 4 4 1 16 1 12 3 15 4 11

.G G G G B B G G R R G G R R+ + − − + + − − + + − − +

(18)

(19)

27 / 34

Experimental results

Experiment environment– Used image

• 24 natural images of Kodak PhotoCD

– Comparing methods• Methods of Gunturk and Lu

– Evaluating quality of demosaicing images1

22 110

1 1( ) 10log 255 ( , ) ( , )MN

y M x NPSNR dB O x y D x y

−

≤ ≤ ≤ ≤

= −

∑ ∑

*

1 1

1 ( , ) ( , )ab Lab Laby M x N

E O x y D x yMN ≤ ≤ ≤ ≤

∆ −∑ ∑

(20)

(21)

where is original color image, andis demosaiced color image.

OD

28 / 34

Table 2. PSNR and ∆Eab measures for edge and smooth regions of demosaiced images.

29 / 34

Fig. 9. (a) original, (b) Gunturk, (c) Lu, (d) proposed interpolation result, (e) Gunturk with Lu’s post-processing, (f) Lu’s post-processing, and (g) proposed post-processing 30 / 34

Fig. 10. (a) original, (b) Gunturk, (c) Lu, (d) proposed interpolation result, (e) Gunturk with Lu’s post-processing, (f) Lu’s post-processing, and (g) proposed post-processing 31 / 34

Fig. 11. (a) original, (b) Gunturk, (c) Lu, (d) proposed interpolation result, (e) Gunturk with Lu’s post-processing, (f) Lu’s post-processing, and (g) proposed post-processing 32 / 34

Table 3. PSNR and ∆Eab measures for a zoom-in of edge and smooth regions of demosaiced images.

33 / 34

Conclusions and future work

New edge-adaptive CFA demosaicing algorithm– Effectively reduce color artifacts

• Both smooth and edge regions

– Combining with any existing image interpolation for reconstructing G channel

– Outperform two recent demosaicing methods

Future research– Developing single-plane reconstruction algorithms

• Reconstructing G channel with minimum interpolation error

34 / 34

A New Edge-Adaptive Demosaicing Algorithm�for Color Filter Arrays���Image and Vision Computing�Vol. 25, No. 9, 2007�Chi-Yi Tsai and Kai-Tai Song��Presented by Jung-Yul ChoiAbstractIntroductionPrevious work슬라이드 번호 5슬라이드 번호 6슬라이드 번호 7Color-difference approach to demosaicing슬라이드 번호 9슬라이드 번호 10슬라이드 번호 11슬라이드 번호 12슬라이드 번호 13슬라이드 번호 14Proposed edge-adaptive demosaicing algorithm슬라이드 번호 16슬라이드 번호 17슬라이드 번호 18슬라이드 번호 19슬라이드 번호 20슬라이드 번호 21슬라이드 번호 22슬라이드 번호 23슬라이드 번호 24Green channel adaptive interpolation슬라이드 번호 26슬라이드 번호 27Experimental results슬라이드 번호 29슬라이드 번호 30슬라이드 번호 31슬라이드 번호 32슬라이드 번호 33Conclusions and future work